A piece of history

FreeBSD jails were introduced in June 2000. They were the first open-source solution for lightweight virtualization, and proved to be foundational to the container revolution that took off later on, preceding the emergence of linux-vserver in October 2001, or LXC containers at the end of 2008.

The jail technology inspired Sun’s engineers, who refined and further elaborated on its concepts through the development of Solaris Zones in 2004, as this talk by Bryan Cantrill amusingly evokes.

While jail may have started as a « chroot on steroids », it has evolved into a full fleged OS containerization solution that has stood the test of time, proven to be robust, and shown to be as relevant as ever today!

Observations

In a nutshell:

  • unlike many other OS virtualization solutions, FreeBSD Jails are very light-weight, and have zero dependency except for a FreeBSD base system ;
  • they are quick to deploy and very versatile, which also means there’s a number of ways one can go about them rather than a one fits all approach, which can be confusing to the newcomer;
  • as a result, there is a ton of well-documented solutions out there, but sorting through 23 years of tutorials to find the best approach to one’s problem can be challenging to say the least;
  • jails’ core concepts and tools are powerful, elegant, and surpringly simple; nevertheless, a succession of jail managers have taken center stage over the years: ezjail, iocage & bastille for the most part, cbsd, pot, raggae too, to forget many lesser-known ones like runj and jailer, which explore interesting concepts (as well as other projects listed in this article;
  • jail management tools add varying degrees of functionality, ease of use and automations; unfortunately, quite a few have seen interrupted development over the years, and while most may remain totally usuable on life support, some have become deprecated as a result;
  • moreover, abstractions don’t usually help fully grasp how software works under the hood, and it so happens that native jails are easy & fun!

Going native

So, here’s what we’re gonna do:

  • avoid the painful task of picking one of the aforementioned projects altogether;
  • create and manage jails manually, using the native toolset provided by FreeBSD;
  • leverage OpenZFS integration for jail deployment:
    • first, by creating a pre-configured jail template for propagation;
    • secondly, by using OpenZFS flags specific to jails if/when relevant (zfs set jailed=on $dataset & zfs jail $jailname $dataset), which will be elaborated upon in a future article;
  • selectively share parts of the host via mount_nullfs in jails, with or without -o ro depending on the case;

Deployment

One dataset per jail

We’re using a dedicated ZFS dataset per jail, so we can use zfs snapshot, zfs clone & zfs send|recv to our advantage when migrating, backuping, replicating & deploying jails:

zfs create zroot/jails/template

Base system install

Let’s install the base system in the newly created dataset:

cd /usr/src && mkdir FreeBSD-14.2-RELEASE && cd $_
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/14.2-RELEASE/base.txz
tar xvf /usr/src/FreeBSD-14.2-RELEASE/base.txz -C /usr/local/jails/template

Alternatively, we can use bsdinstall with the jail option if we want to go through an interactive prompt:

bsdinstall jail /usr/local/jails/template

Basic configuration

Let’s create /etc/jail.conf with common variables and parameters, followed by jail-specific options (alternatively, one can build one config file per jail in /etc/jail.conf.d/:

# Use the rc scripts to start and stop jails.  Mount jail's /dev.
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown jail";
exec.clean;
mount.devfs;

# Dynamic wildcard parameter:
# Base the path off the jail name.
path = "/usr/local/jails/$name";

# jail template
template {
        host.hostname = "template.usc.lan";
        interface = bge0;
        ip4.addr = 10.0.0.10;
        # allow ping
        allow.raw_sockets = 1;
}

We’re using shared networking with the host in this example, but it is possible to virtualize a full network stack inside a jail using vnet. This goes way beyond the scope of this article, but note that additionnal scripts & examples can be found in /usr/share/examples/jails/ to pursue this goal.

Startup

Now, let’s take action:

  1. enable the service: service jail enable …which adds jail_enable="YES" to /etc/rc.conf.
  2. start: jail -c template or service jail start template
  3. list running jails: jls
  4. launch a terminal in the jail: jexec template

Tuning

Further configuration within the jail is touched on in the companion jail tuning article.

Integration

System updates

While the kernel is shared with the host system, each jail runs a separate copy of the base system. However, it has to be updated from the host, using the -j flag or the freebsd-update command:

freebsd-update -j template fetch
freebsd-update -j template install

Userland updates

While userland packages can be installed directly within the jail, one may want to do so from straight from the host as well, removing the need to enter a jail for that sole purpose:

pkg -j template update
pkg -j template install vim zsh

Monitoring

FreeBSD jails being part of the operating system, they benefit from deep integration with some of the system tools as previously shown. Another one of note is top, which can be ran from the host using the -j flag to limit it to a specific jail’s processes.

Propagation

Assuming we’ve gone into the jail and set it up accordingly (installing additionnal packages if needed, setting up proper defaults, etc.), it may be time to deploy other instances!

ZFS cloning

Having used a dedicated zfs dataset, we can simply zfs clone it to duplicate it. Let’s start by taking a snapshot, since one cannot clone a live system:

zfs snap zroot/jails/template@14.2-RELEASE-p2
zfs clone zroot/jails/template@14.2-RELEASE-p2 zroot/jails/test

A word of caution: clones remain tied to the snapshot they were issued from, which can lead into headaches trying to fix inheritance with zfs promote commands.

ZFS sending & receiving

To get rid of this dependency, we can use zfs’s ability to send datasets as a datastream to duplicate our jail, using zfs send & zfs receive, as described in the zfs send/recv companion article.

With no further ado:

zfs send zroot/jails/template@14.2-RELEASE-p2 | \
zfs recv zroot/jails/newjail

Note: don’t forget to create a matching configuration in /etc/jail.conf or /etc/jail.conf.d to get things going.

This is clearly one of the things using a jail management tool can become handy for. We’ll probably detail how to leverage BastilleBSD in a future article, as it can make sense to use such a third-party tool in some instances.

This being said, aren’t you glad you can now create a working isolated environment within minutes, with no additional tools anyhow? 🤓

ZFS within a jail

This is a powerful functionality leveraging FreeBSD’s integration of jail and zfs, allowing a system administrator to delegate the management of specifically designated datasets with the full power of the zfs command to the whoever is managing the jail.

A future companion article will dive into that precisely!

Sharing data from the host

We may want to share a dedicated storage dataset with users of the jail, or a collection of ports in read-only. It’s as simple as using nullfs mounts, which is further explained in the companion jail tuning article.

Documentation

First and foremost, the excellent FreeBSD Mastery: Jails book from Michael W. Lucas comes highly recommended.

Bonus: