A Tale Of Two Memory Leaks In Go

by Baron Schwartz on 15 Jan 2014

A Tale Of Two Memory Leaks

In this post I'll illustrate two ways that I've accidentally caused slow but steady memory consumption in Go programs. The phrase "memory leak" isn't really accurate, but I can't think of a better one. It's more like "using memory in a way that will never be garbage collected."

First - what does such a problem look like? Here's a screenshot of what happened when I introduced one of these bugs into our operating system metrics agent:

Agent Memory Growth

The ticks on the time-scale are in days. The memory growth is slow enough that even if you watch the agent over a few hours after a new release, you might not see it. The sharp jumps downwards are when this agent was released and restarted. You can see that at first its memory usage grows a bit, and then it settles down; this is what we're used to, and the steady climb only becomes clear after most of a day has passed.

Leak 1: Deferred Code In A Loop

The first type of problem I introduced was by (ab)using Go's idiomatic way to clean up resources, the defer statement. If you're not familiar with it, you usually use it thusly:

fh, err := os.Open("/path/to/file.txt")
if err != nil {
   //
}
defer fh.Close()

Imagine this snippet of code in a larger program. The defer allows you to place the closing/cleanup of the resource right next to where it's created. The code is guaranteed to run when the function returns, even if there's a panic. This helps make your code much cleaner. If you've ever written larger programs that open and need to clean up lots of such resources, you'll know what I mean: it's hard to cross-check and ensure that all the t's have been crossed and the i's have been dotted. This idiom avoids the need to cross-check parts of code against other parts.

But I made a subtle mistake: I placed my defer inside a loop in a long-running function:

func somefunc() {
    for {
        // ...
        defer something.Cleanup()
    }
}

The problem here is that the deferred code never executes. Any variable referenced in that code is never garbage-collected. Presto: slow-growing memory usage.

Leak 2: Blocked Goroutines In A Loop

The second type of leak I accidentally created actually looked pretty similar. Go has a go keyword that runs a function in a new goroutine. A goroutine is kind of like a lightweight thread, and the go keyword is a little bit like backgrounding a process in shell with a trailing &.

It's fine to create goroutines in a loop. Goroutines are cheap and they're meant to be used this way:

for {
    go doSomething()
}

The problem I created was because I created goroutines that blocked. They were meant to send a value to a list of channels, and I didn't want them to block, so I "backgrounded" them in goroutines to make them asynchronous. This was the wrong way to do it, as I'll discuss in a bit:

for i := range channels {
    go func() {
        channels[i] <- value
    }()
}

And this worked -- unless the goroutine that was receiving from the channel stopped receiving for any reason. Which they did. Some of them returned from their function and exited; others had errors and exited.

My sending loop, of course, continued merrily on its way, unblocked as intended. But little did I realize that every time it iterated, it created a new goroutine that tried to send to a channel, and blocked forever, thus never exiting. As a result, the program created goroutines forever, and its memory grew steadily.

The correct way to do that, by the way, is with a nonblocking select statement. This was the fix for the bug:

for i := range channels {
    select {
    case channels[i] <- value:
    default:
    }
}

This idiom is discussed at length in the Go documentation.

Conclusion

I hope this "tale of two memory leaks" helps illustrate subtleties in creating and cleaning up resources in Go. Go is pretty obsessed with memory, and exposes that to the programmer. I don't expect that these two problems would be surprising to proficient Go programmers, but perhaps this is useful for those of us who are less experienced.

comments powered by Disqus