Writing and testing Kubernetes webhooks using Kubebuilder v2

Recently, I am leading a project to re-design our on-premise data centers using Kubernetes. Inevitably there are opportunities to develop Kubernetes native applications.

The architecture of Kubernetes is like "hub-and-spoke"; the center of the system is kube-apiserver and all other programs communicate with it.

https://d33wubrfki0l68.cloudfront.net/518e18713c865fe67a5f23fc64260806d72b38f5/61d75/images/docs/post-ccm-arch.png

Ref. https://kubernetes.io/docs/concepts/architecture/cloud-controller/

To customize Kubernetes, kube-apiserver provides the following functions:

This article describes how to implement and test webhooks using Kubebuilder v2. Be warned that the contents are lengthy and meant for professionals.

By implementing webhooks, you can validate and/or mutate resources when resources are created, updated, or deleted as shown below:

https://d33wubrfki0l68.cloudfront.net/af21ecd38ec67b3d81c1b762221b4ac777fcf02d/7c60e/images/blog/2019-03-21-a-guide-to-kubernetes-admission-controllers/admission-controller-phases.png

Ref. https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Implementing webhooks often as easy as just implementing an HTTPS service. But sometimes it can be cumbersome due to the following requirements:

  • Dynamic reloading of TLS certificates
  • Making JSON patches for the response
  • Caching frequently used resources for better performance
  • Testing webhooks with the real kube-apiserver

Kubebuilder helps to implement these requirements with its runtime library controller-runtime. There are some examples:

Unfortunately, the current kubebuilder does not scaffold integration tests for webhooks. To write integration tests, you can use envtest package included in controller-runtime that starts etcd and kube-apiserver.

All other things need to be written manually, which I did. The result is available at GitHub. The rest of this article is the detailed descriptions of the code.

Initializing webhook server

   wh := mgr.GetWebhookServer()
    wh.Host = webhookHost
    wh.Port = webhookPort
    wh.CertDir = certDir

    // NewDecoder never returns non-nil error
    dec, _ := admission.NewDecoder(scheme)
    wh.Register("/mutate", &webhook.Admission{Handler: podMutator{mgr.GetClient(), dec}})

https://github.com/cybozu-go/topolvm/blob/037e972ffe410a3e77513324a07dcee0ed87b652/hook/run.go#L56-L63

Webhook server can only be created within mgr through its GetWebhookServer method. You should initialize the listening address and the certificate directory before registering the first handler.

Starting with v2, kubebuilder relies on cert-manager to prepare certificates for webhooks. For integration tests, you need to prepare the certificate by using OpenSSL or cfssl in a directory.

The initialization function should take the certificate directory and set it to webhooks CertDir field as above.

Markers for webhooks and RBAC

// +kubebuilder:webhook:path=/mutate,mutating=true,failurePolicy=fail,groups="",resources=pods,verbs=create,versions=v1,name=topolvm-hook
// +kubebuilder:rbac:groups="",resources=persistentvolumeclaims,verbs=get;list;watch
// +kubebuilder:rbac:groups=storage.k8s.io,resources=storageclasses,verbs=get;list;watch

Kubebuilder parses special comments called markers in Go source files to generate YAML manifests. The specification of the marker for webhooks is here.

Since my webhook references PersistentVolumeClaims and StorageClasses to process requests, markers for RBAC are also necessary. In addition to get, list and watch must be permitted because the client library of controller-runtime transparently issues List and Watch API when getting resources.

When you edit markers, regenerate YAML manifests by make manifests.

Enabling webhook admission controls

   By("bootstrapping test environment")
    apiServerFlags := envtest.DefaultKubeAPIServerFlags[0 : len(envtest.DefaultKubeAPIServerFlags)-1]
    apiServerFlags = append(apiServerFlags, "--admission-control=MutatingAdmissionWebhook")
    testEnv = &envtest.Environment{
        CRDDirectoryPaths:  []string{filepath.Join("..", "config", "crd", "bases")},
        KubeAPIServerFlags: apiServerFlags,
    }

    var err error
    cfg, err = testEnv.Start()

https://github.com/cybozu-go/topolvm/blob/037e972ffe410a3e77513324a07dcee0ed87b652/hook/suite_test.go#L40-L49

Normally, envtest starts kube-apiserver with all Admission Controllers disabled. Because webhooks are implemented as admission controls, you need to modify the flags for API server.

The above code rewrites the default flags to enable MutatingAdmissionWebhook and starts the integration test environment.

Waiting for webhook server to get ready

   By("running webhook server")
    certDir, err := filepath.Abs("./certs")
    Expect(err).ToNot(HaveOccurred())
    go Run(cfg, "127.0.0.1", 8443, "localhost:8999", certDir, false)

    d := &net.Dialer{Timeout: time.Second}
    Eventually(func() error {
        conn, err := tls.DialWithDialer(d, "tcp", "127.0.0.1:8443", &tls.Config{
            InsecureSkipVerify: true,
        })
        if err != nil {
            return err
        }
        conn.Close()
        return nil
    }).Should(Succeed())

https://github.com/cybozu-go/topolvm/blob/037e972ffe410a3e77513324a07dcee0ed87b652/hook/suite_test.go#L57-L71

To run integration tests, the webhook server has to be started beforehand as a goroutine. Waiting for the webhook server to become ready is important because it takes some time to start accepting requests.

The above code uses Eventually function of Ginkgo to wait. Ginkgo is the testing framework often used in Kubernetes projects.

Installing the webhook configuration

   caBundle, err := ioutil.ReadFile("certs/ca.crt")
    Expect(err).ShouldNot(HaveOccurred())
    wh := &admissionregistrationv1beta1.MutatingWebhookConfiguration{}
    wh.Name = "topolvm-hook"
    _, err = ctrl.CreateOrUpdate(testCtx, k8sClient, wh, func() error {
        failPolicy := admissionregistrationv1beta1.Fail
        urlStr := "https://127.0.0.1:8443/mutate"
        wh.Webhooks = []admissionregistrationv1beta1.Webhook{
            {
                Name:          "hook.topolvm.cybozu.com",
                FailurePolicy: &failPolicy,
                ClientConfig: admissionregistrationv1beta1.WebhookClientConfig{
                    CABundle: caBundle,
                    URL:      &urlStr,
                },
                Rules: []admissionregistrationv1beta1.RuleWithOperations{
                    {
                        Operations: []admissionregistrationv1beta1.OperationType{
                            admissionregistrationv1beta1.Create,
                        },
                        Rule: admissionregistrationv1beta1.Rule{
                            APIGroups:   []string{""},
                            APIVersions: []string{"v1"},
                            Resources:   []string{"pods"},
                        },
                    },
                },
            },
        }
        return nil
    })
    Expect(err).ShouldNot(HaveOccurred())

https://github.com/cybozu-go/topolvm/blob/037e972ffe410a3e77513324a07dcee0ed87b652/hook/mutate_pod_test.go#L29-L60

The last piece of preparation is to add a webhook configuration as MutatingWebhookConfiguration resource. The above code configures a mutating webhook to:

  • Validate certificates using the CA certificate for the server certificate,
  • Set webhook server URL as https://127.0.0.1:8443/mutate, and
  • Add a rule to call the hook upon Pod creation.

Writing test cases

   It("should mutate pod w/ TopoLVM PVC", func() {
        pod := testPod()
        pod.Spec.Volumes = []corev1.Volume{
            {
                Name: "vol1",
                VolumeSource: corev1.VolumeSource{
                    PersistentVolumeClaim: pvcSource("pvc1"),
                },
            },
        }
        err := k8sClient.Create(testCtx, pod)
        Expect(err).ShouldNot(HaveOccurred())

        pod = getPod()
        request := pod.Spec.Containers[0].Resources.Requests["topolvm.cybozu.com/capacity"]
        limit := pod.Spec.Containers[0].Resources.Limits["topolvm.cybozu.com/capacity"]
        Expect(request.Value()).Should(BeNumerically("==", 1<<30))
        Expect(limit.Value()).Should(BeNumerically("==", 1<<30))
    })

Once all the preparation completes, each test case is to simply create a Pod and check the result of mutation. As each test case creates Pod with the same name, the code uses AfterEach function of Ginkgo to delete it after each test.

   AfterEach(func() {
        pod := &corev1.Pod{}
        pod.Name = "test"
        pod.Namespace = "test"
        err := k8sClient.Delete(testCtx, pod, client.GracePeriodSeconds(0))
        Expect(err).ShouldNot(HaveOccurred())
    })

Summary

This article describes how to write webhooks and test it using Kubebuilder v2. Let's enjoy hacking Kubernetes!