Introduction

This blog is part of our ongoing Go language blog series. We publish a new article in the series every three weeks.

We're writing Dolt, the world's first version controlled SQL database. This blog is about different methods to enable stack traces in your Go programs and how we've benefitted from them in Dolt's development.

What are stack traces?

A stack trace is a listing of all the lines of code that a program had been executing when an error occurred. Each function called gets its own line in the listing. A stack track captures context of how the program got into the state it did when an error occurred.

The context stack traces provide is vital not only during local development, where it speeds up tracking down issues, but also when customers encounter problems in production. Having one dramatically simplifies the process of getting to the root cause of a problem.

For this reason, many modern languages include stack traces in error output by default. In Java, this is what happens if an exception makes it to a top-level stack frame without being intercepted, or if you print an exception:

Exception in thread "main" java.util.InputMismatchException
    at java.base/java.util.Scanner.throwFor(Scanner.java:939)
    at java.base/java.util.Scanner.next(Scanner.java:1594)
    at java.base/java.util.Scanner.nextFloat(Scanner.java:2496)
    at com.example.myJavaProject.hello.main(hello.java:12)

In Python, it looks like this:

IndexError: tuple index out of range
*** format_exception:
['Traceback (most recent call last):\n',
 '  File "<doctest default[0]>", line 10, in <module>\n    lumberjack()\n',
 '  File "<doctest default[0]>", line 4, in lumberjack\n    bright_side_of_life()\n',
 '  File "<doctest default[0]>", line 7, in bright_side_of_life\n    return tuple()[0]\n           ~~~~~~~^^^\n',
 'IndexError: tuple index out of range\n']

Lots of other languages support similar functionality, out of the box, by default. When an error happens, you can print it and it tells you where in the code it occurred and the code path it took to get there.

But not Go. Along with JavaScript, Perl, and Rust, Go doesn't provide any stack trace functionality for its error printing by default. We think that's a bad default, and you can do better.

What about panics?

OK, it's true. There is an error mechanism in Go that prints a stack trace out of the box: panic. Here's what it looks like in action:

panic: I'm outta here

goroutine 1 [running]:
main.iPanic(...)
    /tmp/sandbox137813596/prog.go:8
main.main()
    /tmp/sandbox137813596/prog.go:4 +0x39

Any time you call panic() (or dereference a nil pointer, or index a slice out of bounds, or some other things), without a corresponding defer recover() block, your program will halt with this kind of output. Some Go developers have decided this is pretty close to Exceptions in languages like Java, and write code that uses panics for error handling instead of returning errors like normal idiomatic go code. And look, we get the appeal. Nobody enjoys writing go's most famous three lines:

if err != nil {
    return err
}

But the problem is that panicking is just too dangerous. Any panic you forget to recover, in any goroutine, crashes the entire program. We adopted a code base that used panics like exceptions, and rewrote the whole thing to use error returns instead, and it wasn't because it was fun.

So yes, technically if you let a panic crash your application you do get a stack trace. But that's usually a bad thing when it happens, and we don't recommend it. Let's look at other options.

Getting stack traces from error printing

To get useful stack traces out of normal error handling code in Go, you need to do 3 things:

1. Capture a stack trace in the error type when the error is created
2. Implement fmt.Formatter on the error type to print the stack trace
3. Print the error with fmt verb %+v

That's it! Let's look at the final part first, as implemented by the popular testing library testify and its NoError method. When you call require.NoError(t, err), this is what happens behind the scenes:

func NoError(t TestingT, err error, msgAndArgs ...interface{}) bool {
	if err != nil {
		if h, ok := t.(tHelper); ok {
			h.Helper()
		}
		return Fail(t, fmt.Sprintf("Received unexpected error:\n%+v", err), msgAndArgs...)
	}

	return true
}

So if the error you provided has a stack trace, and it implements the fmt.Formatter interface to include the stack trace in its output for %+v, then you get a nice stack trace in your tests when they fail. Great! You can print error stack traces in your application at appropriate places in the same way.

You don't need to implement your own error types to get this functionality. Let's look at some error libraries that make this easy to do.

Error libraries that support stack traces

A lot of people were unsatisfied with Golang's spartan approach to errors in the early days of Go, and many people wrote libraries to fill the gaps. These libraries implemented things like Wrap and Is long before they made it into the standard go libraries. But while Wrap and Is now have full support in the standard library, there's still no official support for stack traces. Which means if you want it, you need to use a third party package, or roll your own.

Here are some of the more popular ones. We currently use src-d/go-errors, but we are looking to migrate or expand to some of the other libraries on this list in the near future.

pkg/errors

pkg/errors is one of the older and probably the most popular alternative to the standard errors library in Go. errors.New captures the stack:

// New returns an error with the supplied message.
// New also records the stack trace at the point it was called.
func New(message string) error {
	return &fundamental{
		msg:   message,
		stack: callers(),
	}
}

And Format prints it:

func (f *fundamental) Format(s fmt.State, verb rune) {
	switch verb {
	case 'v':
		if s.Flag('+') {
			io.WriteString(s, f.msg)
			f.stack.Format(s, verb)
			return
		}
		fallthrough
	case 's':
		io.WriteString(s, f.msg)
	case 'q':
		fmt.Fprintf(s, "%q", f.msg)
	}
}

pkg/errors also has the cool feature of being able to set the stack after the error was created via the WithStack method, which gives you added flexibility.

go-errors/errors

go-errors/errors offers much of the same functionality as pkg/errors, but it's still being actively maintained whereas the latter has been archived.

It also features a way to transform a statically defined error, maybe created with the built-in errors package, into its custom error type with a stack trace. Because the stack trace must be captured at the time the error is returned, this is a great option for people who like to statically define their error types in one central place but still want stack traces for each error returned.

// New makes an Error from the given value. If that value is already an
// error then it will be used directly, if not, it will be passed to
// fmt.Errorf("%v"). The stacktrace will point to the line of code that
// called New.
func New(e interface{}) *Error {
	var err error

	switch e := e.(type) {
	case error:
		err = e
	default:
		err = fmt.Errorf("%v", e)
	}

	stack := make([]uintptr, MaxStackDepth)
	length := runtime.Callers(2, stack[:])
	return &Error{
		Err:   err,
		stack: stack[:length],
	}
}

ztrue/tracerr

ztrue/tracerr is for people who want to take the stack trace concept one step farther and output the actual lines of source code in that trace, complete with colored output, showing you line by line how your error occurred:

This may be overkill for most projects, but it's highly configurable and bound to be exactly what someone reading this is looking for in their personal project.

src-d/go-errors

src-d/go-errors is definitely the least popular library in this list. We know about it because it's used by go-mysql-server, which was originally written by the same organization. Its main innovation is the introduction of what they call an ErrorKind, which is a way to centrally define error types that you instantiate on demand as needed. It looks like this:

ErrColumnNotFound = errors.NewKind("column %q could not be found in any table in scope")

...

err := ErrColumnNotFound.New(columnName) // parameter used in format string above

This is a neat construct that we haven't seen anywhere else. But unfortunately, the src-d organization went out of business a few years ago and the package is no longer being actively maintained. We've flirted with the idea of adopting it, just like we did for go-mysql-server, but we can't decide if that's worthwhile when the above packages exist and are much more broadly adopted.

End to end example

Here's an example of creating an error with a stack trace and then logging its output in the program. We're going to be using the src-d/go-errors library since we have it on hand, but this looks very similar with any of the other libraries above.

package main

import (
	"fmt"

	"gopkg.in/src-d/go-errors.v1"
)

var ExampleError = errors.NewKind("an error happened!")

func main() {
	err :=  foo()
	if err != nil {
		fmt.Printf("Encountered error in main: %+v", err)
	}
}

func foo() error {
	return bar()
}

func bar() error {
	return baz()
}

func baz() error {
	return ExampleError.New()
}

When we run it, we get a nice stack trace printed:

Encountered error in main: an error happened!

main.baz
        C:/Users/zachmu/liquidata/go-workspace/src/github.com/dolthub/dolt/go/scratch/main.go:41
main.bar
        C:/Users/zachmu/liquidata/go-workspace/src/github.com/dolthub/dolt/go/scratch/main.go:37
main.foo
        C:/Users/zachmu/liquidata/go-workspace/src/github.com/dolthub/dolt/go/scratch/main.go:33
main.main
        C:/Users/zachmu/liquidata/go-workspace/src/github.com/dolthub/dolt/go/scratch/main.go:26
runtime.main
        C:/Program Files/Go/src/runtime/proc.go:267
runtime.goexit
        C:/Program Files/Go/src/runtime/asm_amd64.s:1650

Downsides to stack traces in errors

There's one major downside to putting stack traces in your errors: they're expensive. Asking the runtime for a full stack trace and storing is very slow compared to the standard errors.New(), which does this:

// New returns an error that formats as the given text.
// Each call to New returns a distinct error value even if the text is identical.
func New(text string) error {
	return &errorString{text}
}

But this is easily solved: use cheap errors on the hot path. And don't use errors for normal control flow in the first place, that's an antipattern (io.EOF notwithstanding). When you get off the hot path, wrap your hot-path errors with a rich error type as necessary.

Conclusion

Hopefully this blog convinces you of the value of having stacktraces in your error messages. At DoltHub our code base is still very mixed: almost all of the SQL engine code has rich stack trace errors, but many parts of the Dolt storage engine and application code still use normal fmt.Errorf() calls or errors defined with the built-in errors.New(). Changing the code to use rich stack-based errors in more places is an ongoing project for us.

Have questions about Dolt or errors in golang? Join us on Discord to talk to our engineering team and meet other Dolt users.