Sarah Adams
Dec 14, 2015 5 min read

test2doc: Generate Your API Docs

API docs are important. We all know that. They are also painful and tedious to maintain. When your docs aren’t accurate, you get more questions. And everyone loses time.

I’ve always thought of unit tests as a great source of documentation. But the non-Go engineers consuming my API don’t often agree.

Yet most of the same information is there:

request format:
- HTTP method
- URI
- query params
- request headers
- request body
response format:
- status code
- response headers
- response body

All that’s missing from this list is the high-level descriptions, the context.

Enter Go doc.
We can find descriptions of an HTTP handler in its Go doc string, eg:

1
2
3
4


// GetWidget retrieves a single Widget
func GetWidget(w http.ResponseWriter, req *http.Request) {
    // ...
}

And thus began test2doc - automatically generate complete API documentation from your existing Go unit tests + Go doc strings.

Example

Given an HTTP handler func, eg.:

1
2
3
4


// GetWidget retrieves a single Widget
func GetWidget(w http.ResponseWriter, req *http.Request) {
    // ...
}

And a test for this handler func, eg.:

1
2
3
4
5
6


func TestGetWidget(t *testing.T) {
    urlPath := fmt.Sprintf("/widgets/%d", 2)

    resp, err := http.Get(server.URL + urlPath)
    // assert all the things...
}

Test2doc will automatically generate markdown documentation for this endpoint in the API Blueprint format as your tests run, like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# Group widgets

## /widgets/{id}

+ Parameters
    + id: `2` (number)

### Get Widget [GET]
retrieves a single Widget

+ Response 200 

    + Body

            {
                "Id": 2,
                "Name": "Pencil",
                "Role": "Utensil"
            }        

Which you can then parse and host w/ Apiary.io: screenshot

Or use a custom parser and host yourself.

Getting started with test2doc

A big goal of mine as I was writing this project was to limit the amount of additional code needed to get test2doc up and running.

Requirements:

You must have a TestMain for each of the packages you wish to document.
All of the tests in your package must share a single test.Server instance (from test2doc/test package)

3 Code Additions

3 additions, and only to your testing code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


package widgets_test

import (
	"testing"

	"github.com/adams-sarah/test2doc/test"
)

var server *test.Server

func TestMain(m *testing.M) {
	// 1. Tell test2doc how to get URL vars out of your HTTP requests
	//
	//    The 'URLVarExtractor' function must have the following signature:
	//      func(req *http.Request) map[string]string
	//      where the returned map is of the form map[key]value
	test.RegisterURLVarExtractor(myURLVarExtractorFn)


	// 2. You must use test2doc/test's wrapped httptest.Server instead of
	//    the raw httptest.Server, so that test2doc can listen to and
	//    record requests & responses.
	//
	//    NewServer takes your HTTP handler as an argument
	server, err := test.NewServer(router)
	if err != nil {
		panic(err.Error())
	}

	// .. then run your tests as usual
	exitCode := m.Run()


	// 3. Finally, you must tell the wrapped server when you are done testing
	//    so that the buffer can be flushed to an API Blueprint doc file
	server.Finish()

	// note that os.Exit does not respect defers.
	os.Exit(exitCode)
}

Some example URLVarExtractors for different routers:

gorilla/mux:

1
2
3
4
5
6
7


// NOTE: if you are using gorilla/mux, you must set the 
// router's 'KeepContext' to true, so that url parameters 
// can be accessed after the request has been handled.
router.KeepContext = true
    
// Use mux.Vars func as URLVarExtractor
test.RegisterURLVarExtractor(mux.Vars)

julienschmidt/httprouter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


import (
	"net/http"

	"github.com/adams-sarah/test2doc/doc/parse"
	"github.com/julienschmidt/httprouter"
)

// MakeURLVarExtractor returns a func which extracts 
// url vars from a request for test2doc documentation generation
func MakeURLVarExtractor(router *httprouter.Router) parse.URLVarExtractor {
	return func(req *http.Request) map[string]string {
		// httprouter Lookup func needs a trailing slash on path
		path := req.URL.Path
		if !strings.HasSuffix(path, "/") {
			path += "/"
		}

		_, params, ok := router.Lookup(req.Method, path)
		if !ok {
			return nil
		}

		paramsMap := make(map[string]string, len(params))
		for _, p := range params {
			paramsMap[p.Key] = p.Value
		}

		return paramsMap
	}
}

// and then..
test.RegisterURLVarExtractor(MakeURLVarExtractor(router))

How the magic happens

Recording Requests and Responses:

Recording the requests and responses was pretty straight-forward:

Write the HTTP handler’s response initially to an http.ResponseRecorder
Add the response header/body to the documentation
Copy the response back to the original http.ResponseWriter
Continue executing the test

Fetching the handler’s Go doc string:

First, let’s take a look at the http.ResponseWriter interface from Go’s net/http package:

1
2
3
4
5


type ResponseWriter interface {
        Header() Header
        Write([]byte) (int, error)
        WriteHeader(int)
}

test2doc has its own ResponseWriter, which implements http.ResponseWriter, and looks something like this:

1
2
3
4
5


type ResponseWriter struct {
	HandlerInfo HandlerInfo
	URLVars     map[string]string
	W           *httptest.ResponseRecorder
}

test2doc’s ResponseWriter implements the Header and WriteHeader methods by just falling back to those of its embedded httptest.ResponseRecorder:

1
2
3
4
5
6
7


func (rw *ResponseWriter) Header() http.Header {
	return rw.W.Header()
}

func (rw *ResponseWriter) WriteHeader(c int) {
	rw.W.WriteHeader(c)
}

The magic is in the Write implementation. Since every HTTP handler will call the Write method at some point to write out the response, we hijack our ResponseWriter’s Write to inspect the call stack and find our HTTP handler before performing the Write:

1
2
3
4


func (rw *ResponseWriter) Write(b []byte) (int, error) {
	rw.setHandlerInfo()
	return rw.W.Write(b)
}

Here, setHandlerInfo iterates up the call stack (using Go’s runtime package) until it finds a caller whose declaration is inside the package we are testing (in all likelihood, the handler).

Once we have the handler’s function name, we can get the Go doc string using the go/doc package.

Thoughts for the future

I’d like to convert the main types to interfaces, allowing support for formats other than the API Blueprint format.

I’d also like to improve upon the “handler-finding” algorithm (above), to make it more reliably accurate.

Contributions welcome!

Drop me a line at sadams.codes@gmail.com, or adams-sarah on github.

« Generate and Use Free TLS Certificates with Lego Integrating Go in a Yocto-based project »

Gopher Academy Blog