Applications and libraries do not live in a vacuum. This is of course true for Go packages and commands. More often than not, you need your code to interact with legacy applications or old and battle tested libraries.

The standard library of Go provides many facilities for interacting with such entities: encoding/json, encoding/xml or net/rpc. Reach out of the standard library and you have ProtoBuf. All of these facilities meant to arrange for some kind of IPC (Inter-Program Communication) and before Go-1.5, this was the only portable way to do so.

With Go-1.5 and the new execution modes it is now possible to create C shared libraries from a Go package: that’s performed via the -buildmode=c-shared command-line option. As you have seen in libc-hooking-go-shared-libraries, creating a C shared library is not particularly complicated but the process does involve a few steps.

gopy automates the drudgery work of creating a CPython-2 C-extension module out of a Go package: eventually allowing you to write nice concurrent libraries in Go and share them with your CPython friends. Given a Go package, gopy will:

• inspects the Go package
• extracts the exported types, funcs, vars and consts
• creates a cgo package that exports these entities to C
• creates a C extension module, using the CPython-2 API, calling these cgo exported entities
• compiles everything together into a .so shared object

Installation

gopy is a pure-Go command and can thus be installed like so:

1

sh> go get github.com/go-python/gopy

Of course, at runtime, for gopy to be able to actually generate the CPython extension module, it will need:

• the python-dev package (which contains the CPython development headers and libpython.so.2.X)
• pkg-config and the corresponding python2.pc configuration file which holds and describes the correct C incantation commands (-I, -L and -l)
• and, finally, a C compiler.

Example

Consider the following (fictious) github.com/me/hello package:

1
2
3
4
5
6
7
8
9

// hello is a simple package
package hello

import "fmt"

// Hello greets someone.
func Hello(name string) string {
    return fmt.Sprintf("hello %q from Go", name)
}

To create a CPython extension module out of it, you would run:

1
2
3
4
5
6
7

sh> cd somewhere        ## anywhere, under $PYTHONPATH
sh> gopy bind github.com/me/hello
2015/12/07 17:21:36 work: /tmp/gopy-012085634

sh> ll
total 3.2M
-rw-r--r-- 1 binet binet 3.2M Dec  7 17:23 hello.so

The hello CPython extension module can then be imported from your favorite CPython-2 interpreter like any other module:

1
2
3
4

sh> python2
Python 2.7.10 (default, Sep  7 2015, 13:51:49) 
[GCC 5.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

>>> import hello
>>> dir(hello)
['Hello', '__doc__', '__file__', '__name__', '__package__']

>>> print hello.Hello("advent-2015")
hello "advent-2015" from Go

>>> print hello.__doc__
hello is a simple package

>>> print hello.Hello.__doc__
Hello(str name) str

Hello greets someone.

gopy extracted all the Go documentation strings and attached them to their python counterpart. gopy also created a function Hello, translated the Go arguments into their native python counterpart.

If you were curious enough to look under the hood of what gopy generated, you would find:

• a Go file,
• a C header file, and

• a C source file.

  1 2 3 4 5

  sh> ll /tmp/gopy-012085634 total 12K -rw-r--r-- 1 binet binet 4.0K Dec 7 17:23 hello.c -rw-r--r-- 1 binet binet 2.5K Dec 7 17:23 hello.go -rw-r--r-- 1 binet binet 1.7K Dec 7 17:23 hello.h

In the Go file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// Package main is an autogenerated binder stub for package hello.
// gopy gen -lang=go hello
//
// File is generated by gopy gen. Do not edit.
package main

//#cgo pkg-config: python-2.7 --cflags --libs
//#include <stdlib.h>
//#include <string.h>
//#include <complex.h>
import "C"

import (
        "fmt"
        "sync"
        "unsafe"

        "github.com/me/hello"
)

// ...

//export cgo_func_hello_Hello
// cgo_func_hello_Hello wraps hello.Hello
func cgo_func_hello_Hello(name string) (gopy_ret string) {
        _gopy_000 := hello.Hello(name)
        return _gopy_000
}

// buildmode=c-shared needs a 'main'
func main() {}

gopy generates a cgo function which wraps the original Go function hello.Hello and also cgo-export (as cgo_func_hello_Hello) so it is callable from C. The C side looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

/* ... */

#include "Python.h"

// header exported from 'go tool cgo'
#include "hello.h"

#if PY_VERSION_HEX > 0x03000000
#error "Python-3 is not yet supported by gopy"
#endif

/* ... */

/* pythonization of: hello.Hello */
static PyObject*
cpy_func_hello_Hello(PyObject *self, PyObject *args) {
        GoString c_name;
        GoString c_gopy_ret;
        
        if (!PyArg_ParseTuple(args, "O&", cgopy_cnv_py2c_string, &c_name)) {
                return NULL;
        }
        
        
        c_gopy_ret = cgo_func_hello_Hello(c_name);
        
        return Py_BuildValue("O&", cgopy_cnv_c2py_string, &c_gopy_ret);
}

/* functions for package hello */
static PyMethodDef cpy_hello_methods[] = {
        {"Hello", cpy_func_hello_Hello, METH_VARARGS, "Hello(str name) str\n\nHello greets someone.\n"},
        {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
inithello(void)
{
        PyObject *module = NULL;
        
        /* make sure Cgo is loaded and initialized */
        cgo_pkg_hello_init();
        
        module = Py_InitModule3("hello", cpy_hello_methods, "hello is a simple package\n");
        
}

The generated C code uses the CPython API to create a CPython module, fill the ad hoc structures and connect the cgo functions with the CPython infrastructure, together with the python docstrings.

Implementation notes

gopy, like gomobile, has the interesting task of binding two languages together, each of them with a garbage collector. Thus, gopy needs to arrange somehow for the bindings to setup a kind of protocol so that each garbage collector knows when a foreign value is not needed anymore. To that end, gopy notes when a Go value crosses the python boundary and increments a counter so the Go garbage collector will not try to collect it while the python side has a hold on it. On the python side, the C extension module is generated in such a way that the python garbage collector will tell the Go side when that value is not needed anymore, as far as the python side is concerned.

As a consequence, any Go value that somehow crosses the language boundary needs to be allocated on the heap: gopy takes care of that while generating the cgo package.

Schematically, a call from python to a Go function looks like:

1
2
3
4
5
6

// Call sequence (lots of hand-waving)

-> python:    hello.Hello("foo")
 -> cpython:  cpy_func_hello_Hello(...)
  -> cgo:     cgo_func_hello_Hello(...)
   -> go:     hello.Hello(...)

so, while gopy will help bring some of the Go goodies to python, the wrapped entities should not be called inside a very tight loop.

Another interesting point to note is that, as gopy uses cgo, it is subject to the new and more stringent rules that are devised for Go-1.6. As a consequence, some internal parts of gopy will need to be modified to generate code that follows these news. Unfortunately, this probably means another slight performance hit with regard to what could be achieved with Go-1.5.

gopy is currently able to wrap struct types, named types, slice types and array types, as well as the methods which are attached to it. gopy is also clever enough to translate functions using the comma-error idiom into their python equivalent (ie: a function raising an Exception.) Exported vars and consts are also handled: gopy generates “getters” and “setters” for the former and only “getters” for the latter, to preserve the semantics of the original Go package. For slices and arrays, gopy generates code that implements the sequence and buffer protocols. Hence, gopy strives to generate types that look like idiomatic python classes.

Limitations

gopy is still a very young project and many features are still missing. It currently does not support:

• exposing map[T]U types
• exposing chan T types
• exposing interfaces (except error)
• implementing a Go interface from python
• exposing functions or methods taking pointers to values
• wrapping a package which exposes as part of its API types from another package.

Also, currently, gopy generates code only for the CPython-2 API, even if there are no known showstoppers to support CPython-3 or other python VMs.

gopy.py

gopy ships with a little python module – gopy.py – to test and bind interactively any Go package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

>>> import gopy
>>> gopy.load("github.com/me/hello")
gopy> inferring package name...
gopy> loading 'github.com/me/hello'...
2015/12/08 12:29:20 work: /tmp/gopy-706904864
gopy> importing 'github.com/me/hello'
<module 'github.com/me/hello' from '/tmp/gopheracademy/hello.so'>

>>> hello = _
>>> print hello
<module 'github.com/me/hello' from '/tmp/gopheracademy/hello.so'>

>>> print hello.__doc__
hello is a simple package

gopy.py calls the gopy bind command to generate and compile the Go package and then imports the generated C extension module into the current interpreter. This effectively makes every single Go package available from the python interpreter (as long as it is supported by gopy, of course.)

Conclusions

Thanks to the simple rules of the Go language and the packages available with the standard library, it is possible and reasonnably easy to automatically and programmatically:

• inspect a Go package,
• extract the exported API and
• generate bindings for a foreign language, python.

It is not unreasonnable to imagine Go to become the perfect low-level companion language for python, thanks to its quick development cycle and its runtime performance, especially in the concurrent programming space. gopy is a young project, recently made possible thanks to Go-1.5 and there is still a lot of work to support the whole Go language. gopy is released under the BSD-3 license and welcomes bug reports, contributions or new ideas: join us here.

gopy may be just a first step towards a tool suite akin to what SWIG is for C/C++ but for Go. This would help the percolation of Go through “foreign” code bases and strengthen Go’s case as a C replacement. I encourage people interested in this idea to join us at the go-binder organization.