Caching Unstructured Objects using controller-runtime

This is a short blog to let you know that objects represented in unstructured.Unstructured will not be cached by the client returned from manager.Manager by default.

UPDATED on Jan. 9th, 2023
The trick described in this article is no longer needed for controller-runtime version 0.14.0 or later. ref.

This also means that in-memory indexing and retrieving of unstructured objects will not work by default.

To enable caching for unstructured objects, define a custom function to create a client like this:

func NewCachingClient(cache cache.Cache, config *rest.Config, options client.Options, uncachedObjects ...client.Object) (client.Client, error) {
    c, err := client.New(config, options)
    if err != nil {
        return nil, err

    return client.NewDelegatingClient(client.NewDelegatingClientInput{
        CacheReader:       cache,
        Client:            c,
        UncachedObjects:   uncachedObjects,

        // THIS IS THE MAGIC
        CacheUnstructured: true,

and give it when creating manager.Manager as follows:

   mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        NewClient:               NewCachingClient,

Read the rest of the article if you'd like to know more about what's happening under the hood.

Read more

47 Things To Become a Kubernetes Expert

I've been leading a project team for 3 years and have developed a large system around Kubernetes.

In this article, I'd like to share my knowledge and findings of what I think important when developing custom Kubernetes controllers.

Please leave your comments and/or suggestions on my Twitter, if any.

  • API
  • Implementing controllers
  • Components and their collaborations
  • Resources
  • Networking
  • Monitoring
  • Access control
Read more

Use go-grpc-middleware with kubebuilder

Kubernetes and gRPC are technologies to build distributed microservices.

Writing gRPC servers in Go is fun and easy. With go-grpc-middleware, you can add various features to gRPC servers including access logs and Prometheus metrics.

Kubebuilder is a toolkit to develop custom Kubernetes controllers. The main component of kubebuilder is a library called controller-runtime that provides logging and Prometheus metrics as well as Kubernetes client libraries.

When developing a microservice for Kubernetes, these two technologies are often used together. This article describes how. The readers need to have basic knowledge about Go, kubebuilder/controller-runtime, and gRPC.

  • Introducing go-grpc-middleware
  • Problems
  • How to run gRPC server with manager.Manager
  • How to share the same zap logger
  • How to register gRPC metrics with kubebuilder's Registry
  • Conclusion
Read more

An example of using dynamic client of

In this article, I describe how to write a Kubernetes client in Go using the dynamic client in package. During the course, you can learn the following things:

  • The difference between typed clients and the dynamic client.
  • Reading YAML manifests into unstructured.Unstructured.
  • Discovering the REST API endpoint for a Group-Version-Kind.
  • Creating and updating resources using Server Side Apply.

You should have a basic knowledge of Kubernetes and Go programming. The examples in this article depend on

  • Background: Server Side Apply
  • Kubernetes API Basics
    • Mapping between GVK and GVR
  • Go client libraries
    • Typed clients
    • Dynamic client
  • Using the dynamic client to implement SSA

Background: Server Side Apply

Recently, I wrote a program that applies Kubernetes resources using Server Side Apply. Server Side Apply, or SSA, is a new way to create or update resources in Kubernetes API server added as a beta feature to Kubernetes 1.16.

One of the advantage of SSA is that it introduces better patching strategy than Strategic Merge Patch. For example, if a Service has two ports sharing the same port number but with different protocol, Strategic Merge Patch could not identify which port should be updated because it uses port as the key.

apiVersion: v1
kind: Service
  name: mydns
    app: mydns
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
Read more

Writing and testing Kubernetes webhooks using Kubebuilder v2

Recently, I am leading a project to re-design our on-premise data centers using Kubernetes. Inevitably there are opportunities to develop Kubernetes native applications.

The architecture of Kubernetes is like "hub-and-spoke"; the center of the system is kube-apiserver and all other programs communicate with it.


To customize Kubernetes, kube-apiserver provides the following functions:

This article describes how to implement and test webhooks using Kubebuilder v2. Be warned that the contents are lengthy and meant for professionals.

Read more

Making well-behaved programs in Go with cybozu-go/cmd

We at are using Go for variety of tasks.

In early days when we started using Go, there were only few in-house rules to create commands. As a result, we faced several difficulties:

  • Some servers did not record access logs.
  • Some programs did not record command execution logs.
  • No common spec to handle SIGINT or SIGTERM.
  • Some programs did not reopen log files, making log rotation difficult.
  • Inconsistent log formats.
Read more