How to hack Go's http.Server correctly

Go

One of my recent work was a design and implementation of a framework for Go programs. It is open-sourced at GitHub https://github.com/cybozu-go/cmd .

During the work, I have tackled how to enhance Go's http.Server by adding access logging and ability to stop gracefully. This article summarizes my findings and tricks used in github.com/cybozu-go/cmd package.

Overriding http.ResponseWriter

We wanted to add access log transparently to http.Server. Transparency is important for existing applications that use the standard Go HTTP API.

The access log should include these information:

  • Response data size.
  • Elapsed time for the request.
  • HTTP status code.

cmd.HTTPServer wraps http.Server and overrides its Handler to record access log automatically. To collect information for response data size and status code, the overriding handler replaces http.ResponseWriter with:

type logResponseWriter struct {
    StdResponseWriter
    status int
    size   int64
}

logResponseWriter overrides Write, WriteHeader, and a few other methods to collect information in status and size. The most critical part is, however, StdResponseWriter.

// StdResponseWriter is the interface implemented by
// the ResponseWriter from http.Server.
//
// HTTPServer's ResponseWriter implements this as well.
type StdResponseWriter interface {
    http.ResponseWriter
    io.ReaderFrom
    http.Flusher
    http.CloseNotifier
    http.Hijacker
    WriteString(data string) (int, error)
}

StdResponseWriter is defined to mimic ResponseWriter from the original http.Server since the original implements more than defined in ResponseWriter. As you can see in the above snippet, the original response writer implements io.ReaderFrom, http.CloseNotifier, http.Hijacker and so on. If logResponseWriter used the plain http.ResponseWriter, all these optional but important interfaces would be lost.

By defining and using StdResponseWriter, our framework can provide the same power and functions as the original implementation.

Graceful stop for http.Server

Generally speaking, a network server can be stopped gracefully in these steps:

  1. Stop listening.
  2. Wait for all connections to close.
  3. Exit.

Since HTTP/1.1 allows keep-alive connections, the steps become:

  1. Stop listening.
  2. Disable keep-alive by calling http.Server.SetKeepAlive(false).
  3. Wait for all connections to close.
  4. Exit.

Simple, so far. The difficult part is how to interrupt idle connections already waiting in http.Server for next requests quickly. If we failed to interrupt them, step 3 would take as long as http.Server.ReadTimeout duration*1.

To trace connection status changes, http.Server provides ConnState callback:

        // ConnState specifies an optional callback function that is
        // called when a client connection changes state. See the
        // ConnState type and associated constants for details.
        ConnState func(net.Conn, ConnState)

By using a custom ConnState handler, we can maintain a list of idle connections. Interrupting idle connections can be done just by calling conn.SetReadDeadline(time.Now()).

The final implementation is a bit more complicated to cope with some race conditions: https://github.com/cybozu-go/cmd/blob/410d0a67ddf281b023d62b3985a5ecb86af4f1ad/http.go#L214-L254

Conclusion

github.com/cybozu-go/cmd provides an enhanced http.Server without losing the original power and functions with tricks described in this article.

If you are interested, please try it out!

*1:http.Server.ReadTimeout works also as keep-alive timeout in the current implementation.