GopherAcademy
Tommi Virtanen
Dec 11, 2014 10 min read

Writing file systems in Go with FUSE

Motivation

Some time ago, I decided I wanted to solve my own storage needs better, and I realized that I can’t just rely on synchronizing files. I needed a filesystem that combines the best of three worlds: local files, network file systems, and file synchronization. This project is called Bazil, as in bazillion bytes.

To make Bazil possible, I needed to be able to easily write a filesystem in Go. And now you can, too, with bazil.org/fuse.

What we’ll build today is an example Go application that serves a Zip archive as a filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

$ unzip -v archive.zip
Archive:  archive.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
       0  Stored        0   0% 2014-12-11 04:03 00000000  buried/
       0  Stored        0   0% 2014-12-11 04:03 00000000  buried/deep/
       5  Stored        5   0% 2014-12-11 04:03 2efcceec  buried/deep/loot
      13  Stored       13   0% 2014-12-11 04:03 f4247453  greeting
--------          -------  ---                            -------
      18               18   0%                            4 files
$ zipfs archive.zip mnt &
$ tree mnt
mnt
├── buried
│   └── deep
│       └── loot
└── greeting

2 directories, 2 files
$ cat mnt/greeting
hello, world

FUSE

FUSE (Filesystem In Userpace) is a Linux kernel filesystem that sends the incoming requests over a file descriptor to userspace. Historically, these have been served with a C library of the same name, but ultimately FUSE is just a protocol. Since then, the protocol has been implemented for other platforms such as OS X, FreeBSD and OpenBSD.

bazil.org/fuse is a reimplementation of that protocol in pure Go.

Structure of Unix filesystems

Unix filesystems consist of inodes (“index nodes”). These nodes are files, directories, etc. Directories contain directory entries (dirent) that point to child inodes. A directory entry is identified by its name, and carries very little metadata. The inode manages both the metadata (including things like access control) and the content of the file.

Open files are identified in userspace with file descriptors, which are just safe references to kernel objects known as handles.

Go API

Our FUSE library is split into two parts. The low-level protocol is in bazil.org/fuse while the higher-level, optional, state machine keeping track of object lifetimes is bazil.org/fuse/fs.

Each file system has a root entry. The interface fs.FS has a method Root that returns an fs.Node.

To access a file (see its metadata, open it, etc), the kernel looks it up by name by sending a fuse.LookupRequest to the FUSE server, stating the parent directory and basename. This request is served by a Lookup method on the parent fs.Node. The method returns an fs.Node, and the result is cached in the kernel and reference counted. Dropping a cache entry sends a ForgetRequest, and when the reference count reaches zero, Forget gets called.

Files are renamed with Rename, deleted with Remove, and so on.

Kernel file handles are created for example by opening a file. Opening an existing file sends an OpenRequest, you guessed it, served by Open. All methods creating new handles return a Handle. Handles are closed by a combination of Flush and Release.

The default Open action, if the method is not implemented, is to use the fs.Node also as a Handle; this tends to work well for stateless read-only files.

Reads from a Handle are served by Read, writes with Write, and apart from all the extra data available these look similar to io.ReaderAt and io.WriterAt. Note that file size changes via Setattr, not based on Write, and Attr needs to return the correct Size.

Listing a directory happens by reading an open file handle that is a directory. Instead of file contents, the read returns marshaled directory entries. The ReadDir method implements a slightly higher-level API, where you return a slice of directory entries.

And so on. Learning to write a file system requires a decent understanding of the kernel data structures and their state changes on an abstract level, but the actual Go parts of it are quite simple. So let’s dive into the code.

zipfs

As our example project, we’ll write a filesystem that shows a read-only view of the contents of a Zip archive.

The full source code is available at https://github.com/bazillion/zipfs

Skeleton

Let’s start with a skeleton with a argument parsing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

package main

import (
	"archive/zip"
	"flag"
	"fmt"
	"io"
	"log"
	"os"
	"path/filepath"
	"strings"

	"bazil.org/fuse"
	"bazil.org/fuse/fs"
)

// We assume the zip file contains entries for directories too.

var progName = filepath.Base(os.Args[0])

func usage() {
	fmt.Fprintf(os.Stderr, "Usage of %s:\n", progName)
	fmt.Fprintf(os.Stderr, "  %s ZIP MOUNTPOINT\n", progName)
	flag.PrintDefaults()
}

func main() {
	log.SetFlags(0)
	log.SetPrefix(progName + ": ")

	flag.Usage = usage
	flag.Parse()

	if flag.NArg() != 2 {
		usage()
		os.Exit(2)
	}
	path := flag.Arg(0)
	mountpoint := flag.Arg(1)
	if err := mount(path, mountpoint); err != nil {
		log.Fatal(err)
	}
}

Mounting is a bit cumbersome due to OSXFUSE behaving very differently from Linux; there are several stages where errors may show up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

func mount(path, mountpoint string) error {
	archive, err := zip.OpenReader(path)
	if err != nil {
		return err
	}
	defer archive.Close()

	c, err := fuse.Mount(mountpoint)
	if err != nil {
		return err
	}
	defer c.Close()

	filesys := &FS{
		archive: &archive.Reader,
	}
	if err := fs.Serve(c, filesys); err != nil {
		return err
	}

	// check if the mount process has an error to report
	<-c.Ready
	if err := c.MountError; err != nil {
		return err
	}

	return nil
}

Filesystem

On to the actual file system. We just hold a pointer to the zip archive:

1
2
3

type FS struct {
	archive *zip.Reader
}

And we need to provide the Root method:

1
2
3
4
5
6
7
8

var _ fs.FS = (*FS)(nil)

func (f *FS) Root() (fs.Node, fuse.Error) {
	n := &Dir{
		archive: f.archive,
	}
	return n, nil
}

Directories

Zip files contain a list of files, but typical zip archivers include entries for the directories, with a name ending in a slash. We rely on this behavior later.

Let’s define our Dir type, and implement the mandatory Attr method. We use the *zip.File to serve directory metadata.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

type Dir struct {
	archive *zip.Reader
	// nil for the root directory, which has no entry in the zip
	file *zip.File
}

var _ fs.Node = (*Dir)(nil)

func zipAttr(f *zip.File) fuse.Attr {
	return fuse.Attr{
		Size:   f.UncompressedSize64,
		Mode:   f.Mode(),
		Mtime:  f.ModTime(),
		Ctime:  f.ModTime(),
		Crtime: f.ModTime(),
	}
}

func (d *Dir) Attr() fuse.Attr {
	if d.file == nil {
		// root directory
		return fuse.Attr{Mode: os.ModeDir | 0755}
	}
	return zipAttr(d.file)
}

Directory entry lookup

For our filesystem to contain anything useful, we need to be able to find entries by name. We just iterate over the zip entries, matching paths:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

var _ = fs.NodeRequestLookuper(&Dir{})

func (d *Dir) Lookup(req *fuse.LookupRequest, resp *fuse.LookupResponse, intr fs.Intr) (fs.Node, fuse.Error) {
	path := req.Name
	if d.file != nil {
		path = d.file.Name + path
	}
	for _, f := range d.archive.File {
		switch {
		case f.Name == path:
			child := &File{
				file: f,
			}
			return child, nil
		case f.Name[:len(f.Name)-1] == path && f.Name[len(f.Name)-1] == '/':
			child := &Dir{
				archive: d.archive,
				file:    f,
			}
			return child, nil
		}
	}
	return nil, fuse.ENOENT
}

Files

Our Lookup above returned File types when the matched entry did not end in a slash. Let’s define type File, using the same zipAttr helper as for directories:

1
2
3
4
5
6
7
8
9

type File struct {
	file *zip.File
}

var _ fs.Node = (*File)(nil)

func (f *File) Attr() fuse.Attr {
	return zipAttr(f.file)
}

Files are not very useful unless you can open them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

var _ = fs.NodeOpener(&File{})

func (f *File) Open(req *fuse.OpenRequest, resp *fuse.OpenResponse, intr fs.Intr) (fs.Handle, fuse.Error) {
	r, err := f.file.Open()
	if err != nil {
		return nil, err
	}
	// individual entries inside a zip file are not seekable
	resp.Flags |= fuse.OpenNonSeekable
	return &FileHandle{r: r}, nil
}

Handles

1
2
3
4
5

type FileHandle struct {
	r io.ReadCloser
}

var _ fs.Handle = (*FileHandle)(nil)

We hold an “open file” inside our handle. In this case, it’s just a helper type in archive/zip, but in another filesystem this might be a *os.File, a network connection, or such. We should be careful to close them:

1
2
3
4
5

var _ fs.HandleReleaser = (*FileHandle)(nil)

func (fh *FileHandle) Release(req *fuse.ReleaseRequest, intr fs.Intr) fuse.Error {
	return fh.r.Close()
}

And then let’s handle actual Read operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

var _ = fs.HandleReader(&FileHandle{})

func (fh *FileHandle) Read(req *fuse.ReadRequest, resp *fuse.ReadResponse, intr fs.Intr) fuse.Error {
	// We don't actually enforce Offset to match where previous read
	// ended. Maybe we should, but that would mean'd we need to track
	// it. The kernel *should* do it for us, based on the
	// fuse.OpenNonSeekable flag.
	buf := make([]byte, req.Size)
	n, err := fh.r.Read(buf)
	resp.Data = buf[:n]
	return err
}

Readdir

At this point, our files are accessible by cat and such, but you need to know their names. Let’s add support for ReadDir:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

var _ = fs.HandleReadDirer(&Dir{})

func (d *Dir) ReadDir(intr fs.Intr) ([]fuse.Dirent, fuse.Error) {
	prefix := ""
	if d.file != nil {
		prefix = d.file.Name
	}

	var res []fuse.Dirent
	for _, f := range d.archive.File {
		if !strings.HasPrefix(f.Name, prefix) {
			continue
		}
		name := f.Name[len(prefix):]
		if name == "" {
			// the dir itself, not a child
			continue
		}
		if strings.ContainsRune(name[:len(name)-1], '/') {
			// contains slash in the middle -> is in a deeper subdir
			continue
		}
		var de fuse.Dirent
		if name[len(name)-1] == '/' {
			// directory
			name = name[:len(name)-1]
			de.Type = fuse.DT_Dir
		}
		de.Name = name
		res = append(res, de)
	}
	return res, nil
}

Testing zipfs

Prepare a zip file:

1
2
3
4

$ mkdir -p data/buried/deep
$ echo hello, world >data/greeting
$ echo gold >data/buried/deep/loot
$ ( cd data && zip -r -q ../archive.zip . )

Mount it:

1
2

$ mkdir mnt
$ zipfs archive.zip mnt &

Lookup directory entries:

1
2
3
4

$ ls -ld mnt/greeting
-rw-r--r-- 1 root root 13 Dec 11  2014 mnt/greeting
$ ls -ld mnt/buried
drwxr-xr-x 1 root root 0 Dec 11  2014 mnt/buried

Read file contents:

1
2
3
4

$ cat mnt/greeting
hello, world
$ cat mnt/buried/deep/loot
gold

Readdir (the “total 0” is not correct, but that doesn’t matter):

1
2
3
4
5
6
7

$ ls -l mnt
total 0
drwxr-xr-x 1 root root  0 Dec 11  2014 buried
-rw-r--r-- 1 root root 13 Dec 11  2014 greeting
$ ls -l mnt/buried
total 0
drwxr-xr-x 1 root root 0 Dec 11  2014 deep

Unmount (for OS X, use umount mnt):

1

$ fusermount -u mnt

That’s it! For a longer and more featureful examples to read, see https://github.com/bazillion/bolt-mount (screencast of a code walkthrough) and all of the projects importing fuse.

Resources

« goquery: a little like that j-thing Easy Docker Deployment with Hooks and Captain Hook »