Embedding assets in Go

One of the things chef-runner can do for you is to install Chef on a machine before provisioning it. This allows you to set up bare servers that have nothing installed but the base operating system. For this feature, chef-runner used to download the Omnibus installer (better known as install.sh script) to a local folder before copying it to the target machine. There it will be executed to install the Chef version you asked for.

For the latest release of chef-runner, I wanted to change the mechanism a bit. Instead of downloading the installer script from the Internet, I thought it would be better to embed the script and make it part of chef-runner’s source code. I mainly did it for the following reasons:

• Simplicity. The code ends up being simpler. No download logic. No clever caching.
• Transparency. Always use the same script that is checked into version control.
• Speed. Don’t have to download the script again for each project.

On the other hand, I decided to ignore the drawback that you might not benefit from improvements to the installer immediately, simply because Chef Inc. updates the script so rarely.

chef-runner is written in Go. I knew that it is possible to embed assets in Go and this seemed like the perfect opportunity to get familiar with the tooling. Here’s how I ended up doing it using a combination of go-bindata and go generate.

go-bindata

go-bindata converts any text or binary file into Go source code, which is useful for embedding data into Go programs. For example, Inspeqtor uses it to embed email templates.

I used go-bindata to add the Omnibus installer as an asset to chef-runner’s omnibus package, as shown below:

$ cd chef/omnibus/
$ mkdir assets
$ curl https://www.chef.io/chef/install.sh > assets/install.sh
$ go-bindata -pkg omnibus -o assets.go assets/

Afterwards I had to adapt chef/omnibus/omnibus.go to use the asset directly instead of downloading it. Asset data can be accessed via the Asset function, which is included in the generated assets.go source file. The resulting code ended up looking something similar to what you see here:

// chef/omnibus/omnibus.go

package omnibus

type Installer struct {
	ChefVersion string
	SandboxPath string
	RootPath    string
	Sudo        bool
}

func (i Installer) writeOmnibusScript() error {
        script := path.Join(i.SandboxPath, "install.sh")
        log.Debugf("Writing Omnibus script to %s\n", script)
        data, err := Asset("assets/install.sh")
        if err != nil {
                return err
        }
        return ioutil.WriteFile(script, []byte(data), 0644)
}

That’s really all I had to do as far as embedding is concerned.

go generate

While tools like go-bindata are valuable on their own, you still need to integrate them into your build process. For this, you might consider using a build tool like Make to glue all pieces together. But wouldn’t it be great if Go itself would provide the ability to automatically generate source code? Well, it does!

Go 1.4 introduces a new command, go generate, to automate the generation of source code before compilation. The tool works by scanning Go source code for special comments that define commands to run. This way you can declare build instructions in your code, keeping everything together in a nice way.

This is the comment I added to chef-runner’s omnibus package:

// chef/omnibus/omnibus.go

package omnibus

//go:generate go-bindata -pkg $GOPACKAGE -o assets.go assets/

Now, when running go generate, it will pick up the command and execute go-bindata with the specified parameters ($GOPACKAGE will be replaced with the package name, i.e., “omnibus”):

$ go generate -x ./chef/omnibus
go-bindata -pkg omnibus -o assets.go assets/

The -x flag causes go generate to print commands as they are executed. The one command shown above should look familiar.

As with most Go tools, you can run go generate ./... to process all packages of your project at once. To learn more about go generate, consult go help generate or check out this article on the Go blog.

Caveat

There is one thing you need to be aware of when using go generate. The tool isn’t integrated with go get, as one might expect. Because of that, your project will only be “go gettable” if you check in all sources created by go generate. That’s why I put the mentioned assets.go file under version control.

All in all, embedding assets in Go is straight forward thanks to existing tooling.

Update: I wrote a more detailed article based on this post.