Zero downtime API in Golang
If we start a new application we should take in my that our application should be easily upgradable and distributable. Some applications require maximum uptime. This application can be, for example, an API that can be used by multiple clients who expect that the API is always alive.
Recently I have encountered a task, where I needed to develop an API, that has maximum uptime during upgrading binaries. I found two usable solutions, that I would like to share with you. I will try to sum up everything so it can serve as some basepoint for everyone interested in this topic. Writing a blog post about this problem also helps me so I can come back later and quickly read what I have discovered to prevent researching everything from scratch.
Everything that we will talk about is mostly focused on Linux, but it should also work on any UNIX platform. In the blog post, I assume that you know a basic of file descriptors, signals and diferences between threads and processes. These examples are focused on Golang primarily.
During the research, I found two usable solutions that we can use to upgrade our binaries of the API. There are probably more of them, but these are the most used ones. Let’s point the principle of each one.
- We can take an advantage of socket options namely SO_REUSEPORT. In Golang we can set options to our listener (as can be seen in the following code example), with these options we can start multiple web servers that use the same port.
- When we start a new process in Golang, you can specify file descriptors that can be inherited. We can share our opened socket with our child so our child can start where we left off.
The first variant is the most simple to implement. The catch is only in on how to persuade Golang’s TCP listener to apply our socket options. We can see it in the following code snippet.
We apply both SO_REUSEADDR and SO_REUSEPORT, the difference between these two is nicely described in this Stackoverflow answer. Normally you cannot listen on a port that is already used by another application. But when we specify these socket options we can. When we have multiple applications listening on the same port, our kernel will then “randomly” load balance incoming requests between these applications. Everything can be seen in the code here.
The second variant is rather more complex but can be used in multiple scenarios, not only with sockets. It can be used with any kind of file descriptors like pipes, files, etc. In Golang we have many options on how to spawn another process (Do not mix terms like process and thread, the process can be simply explained as another running application. Furthermore processes do not share RAM like threads.). In particular, let’s take a look at StartProcess func in the os package. This func has an option of ProcAttr where we can provide opened file descriptors for our child process. This means that our child process will not create a new listener, it will just take it from it’s parent. Everything can be seen in the code here.
Not let’s test both examples! Both options are hosted on a GitHub. Again let’s start with the first one. The code for the first example is in the socket-option-version. If we compile it with go build -o zero ./main.go and start it as ./zero we can see the output. Let’s send an HTTP request to the endpoint to verify functionality. We can perform the binary reload with a Linux signal SIGUSR2 . The SIGUSR2 is widely adopted for binary reloads. We can send the signal from another terminal like pkill -SIGUSR2 zero, afterwards we can see that the application “closed” and we are able to use the terminal again. Furthermore, we can send another HTTP request and we get the response. The log is printed to the terminal because we shared the stdout and stderr with our child process, this means that any output of our child process will be printed to the terminal.
The second example is located in the folder inherit-version. We can compile it with go build -o zero ./main.go and start it as ./zero we can see the output. And we get the same results!
