Use go-grpc-middleware with kubebuilder

Go Kubernetes Programming gRPC

Kubernetes and gRPC are technologies to build distributed microservices.

Writing gRPC servers in Go is fun and easy. With go-grpc-middleware, you can add various features to gRPC servers including access logs and Prometheus metrics.

Kubebuilder is a toolkit to develop custom Kubernetes controllers. The main component of kubebuilder is a library called controller-runtime that provides logging and Prometheus metrics as well as Kubernetes client libraries.

When developing a microservice for Kubernetes, these two technologies are often used together. This article describes how. The readers need to have basic knowledge about Go, kubebuilder/controller-runtime, and gRPC.

Introducing go-grpc-middleware

go-grpc-middleware is an umbrella project for useful gRPC interceptors for Go. You can easily add access logging, Prometheus metrics, and OpenTracing tags to your gRPC server with go-grpc-middleware.

The following code illustrates how to extract fields from request parameters and log requests using zap.

import (
    grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
    grpc_zap "github.com/grpc-ecosystem/go-grpc-middleware/logging/zap"
    grpc_ctxtags "github.com/grpc-ecosystem/go-grpc-middleware/tags"
    "go.uber.org/zap"
    "google.golang.org/grpc"
)

// extract fields from request parameters
func extractFields(fullMethod string, req interface{}) map[string]interface{} {
    args, ok := req.(*SomeArgumentType)
    if !ok {
        return nil
    }

    ret := make(map[string]interface{})
    if args.Foo != "" {
        ret["foo"] = args.Foo
    }
    ret["bar"] = args.Bar
    return ret
}

func makeServer(log *zap.Logger) *grpc.Server {
    return grpc.NewServer(grpc.UnaryInterceptor(
        grpc_middleware.ChainUnaryServer(
            grpc_ctxtags.UnaryServerInterceptor(grpc_ctxtags.WithFieldExtractor(extractFields)),
            grpc_zap.UnaryServerInterceptor(log),
        ),
    ))
}

This server will log requests with grpc.request.foo and grpc.request.bar fields. If zap is configured to output JSON, it should look like:

{
  "grpc.code": "OK",
  "grpc.method": "Add",
  "grpc.request.foo": "some message",
  "grpc.request.bar": 10,
  "grpc.service": "your.service",
  "grpc.start_time": "2020-08-30T10:18:49Z",
  "grpc.time_ms": 102.34100341796875,
  "level": "info",
  "msg": "finished unary call with code OK",
  "peer.address": "@",
  "span.kind": "server",
  "system": "grpc",
  "ts": 1598782729.4605289
}

Problems

Kubebuilder generates code that assumes the main function runs only manager.Manager. Custom controllers, admission webhooks, and the metrics exporter for Prometheus are all run under the manager. The first problem is: how can we do the same for gRPC server?

The second and third problems are the duplication of features between kubebuilder and go-grpc-middleware.

Both uses zap as a logging library, but kubebuilder abstracts it in logr interface and does not expose raw zap structs. The second problem is: how can we share the same zap logger?

To export metrics for Prometheus, kubebuilder defines a custom Registry whereas go-grpc-middleware's example uses the builtin Registry as follows:

myServer := grpc.NewServer(
    // snip...
    grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(
        grpc_ctxtags.UnaryServerInterceptor(),
        grpc_prometheus.UnaryServerInterceptor,  // uses the builtin registry in the prometheus client library

So the last problem is: how can we register gRPC metrics with kubebuilder's Registry?

How to run gRPC server with manager.Manager

manager.Manager can run arbitrary processes that implements Runnable interface. The following example runs gRPC server under the manager.

import (
    "google.golang.org/grpc"
    ctrl "sigs.k8s.io/controller-runtime"
)

// GRPCRunner implements `manager.Runnable`.
type GRPCRunner {
    server   *grpc.Server
    listener net.Listener
}

// Start the gRPC server.  It will be gracefully shutdown when `ch` is closed.
func (r GRPCRunner) Start(ch <-chan struct{}) error {
    go func() {
        <-ch
        r.server.GracefulStop()
    }()

    return r.server.Serve(r.listener)
}

func main() {
    mgr, _ := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{...}
    l, _ := net.Listen("unix", "/...")

    grpcServer := grpc.NewServer()
    r := GRPCRunner{server: gRPCServer, listener: l}
    mgr.Add(r)

    mgr.Start(ctrl.SetupSignalHandler())
}

How to share the same zap logger

Normally, kubebuilder generates code like this:

import "sigs.k8s.io/controller-runtime/pkg/log/zap"

func main() {
    // ...
    ctrl.SetLogger(zap.New(zap.UseDevMode(true)))
}

The problem here is that zap.New from controller-runtime does not return the raw *zap.Logger. Instead, it returns an object wrapped in logr.Logger interface.

We can rewrite the code so that it creates and shares the same *zap.Logger as follows:

import (
    "github.com/go-logr/zapr"
    grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
    grpc_zap "github.com/grpc-ecosystem/go-grpc-middleware/logging/zap"
    grpc_ctxtags "github.com/grpc-ecosystem/go-grpc-middleware/tags"
    "google.golang.org/grpc"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
)

func main() {
    // zap.NewRaw creates *zap.Logger
    zapLogger := zap.NewRaw()
    defer zapLogger.Sync()

    // Wrap the raw logger with zapr.NewLogger
    ctrl.SetLogger(zapr.NewLogger(zapLogger))

    // Use the same logger for gRPC
    grpcLogger := zapLogger.Named("grpc")
    grpcServer := grpc.NewServer(grpc.UnaryInterceptor(
        grpc_middleware.ChainUnaryServer(
            grpc_ctxtags.UnaryServerInterceptor(),
            grpc_zap.UnaryServerInterceptor(gRPCLogger),
        ),
    ))
}

How to register gRPC metrics with kubebuilder's Registry

Kubebuilder exports various metrics registered with metrics.Registry. To register gRPC metrics with metrics.Registry, write code like this:

import (
    grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
    grpc_ctxtags "github.com/grpc-ecosystem/go-grpc-middleware/tags"
    grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus"
    "google.golang.org/grpc"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

func main() {
    grpcMetrics := grpc_prometheus.NewServerMetrics()
    metrics.Registry.MustRegister(grpcMetrics)

    grpcServer := grpc.NewServer(grpc.UnaryInterceptor(
        grpc_middleware.ChainUnaryServer(
            grpc_ctxtags.UnaryServerInterceptor(grpc_ctxtags.WithFieldExtractor(fieldExtractor)),
            grpcMetrics.UnaryServerInterceptor(),
        ),
    ))

    // register your service

    // after all services are registered, initialize metrics.
    grpcMetrics.InitializeMetrics(grpcServer)
}

This combined with manager.Manager exports metrics including gRPC's.

Conclusion

Now we can use go-grpc-middleware together with kubebuilder. You can find a real world example in github.com/cybozu-go/coil/v2/runners/coild_server.go.