USC2025+SE2 — Backups for the people!

We have started deploying a new backup server, levering the zfs filesystem together with FreeBSD jails 🤓

FreeBSD jails: ZFS inside

So, we’ve seen how to create a native jail using FreeBSD’s toolset, and we’ve fine-tuned a few of its settings, including mounting select directories from the host into the jail.

Is that really enough though? 🙃

ZFS inside

We want zfs inside our jail, period!

But why?

Since we use a dedicated zfs dataset per jail, isn’t that enough? Well, dataset management (and anything disk-related) is handled on the host.

Practically speaking, this means that the root user inside the jail cannot alter dataset properties, nor create new ones.

FreeBSD jails: system tuning

So, we’ve seen how to create a native jail using FreeBSD’s toolset. Meaning we have a brand-new system to configure!

Jail characteristics

Some jail-related specificities:

• each jail runs with the host’s FreeBSD kernel;
• as a result, a jail cannot run a newer OS version than the host system;
• network is shared with the host by default, though the creation of vnet jails allows for virtualizing the entire network stack;
• a number of actions are performed from the host and are either impossible or redundant within each jail; this is obviously the case for anything hardware-related, such as physical disks’ management;
• one may want to share & centralize a number of operations such as logs;
• one may want to access some parts of the host’s filesystem from within a jail;

Jail configuration

Basic Setup

Let’s copy /etc/resolv.conf & /etc/localtime from the host into the jail, so that it can issue DNS requests, and most importantly be on time ;)

FreeBSD jails: going native

A piece of history

FreeBSD jails were introduced in June 2000. They were the first open-source solution for lightweight virtualization, and proved to be foundational to the container revolution that took off later on, preceding the emergence of linux-vserver in October 2001, or LXC containers at the end of 2008.

The jail technology inspired Sun’s engineers, who refined and further elaborated on its concepts through the development of Solaris Zones in 2004, as this talk by Bryan Cantrill amusingly evokes.

ZFS: working with send/recv

Usage

Basics

The following example aims at illustrating the core concept of zfs send/recv:

1. Let’s create a dataset via zfs create storage/test-source;
2. Next, we’ll take a snapshot, since only snapshots of a live filesystem can be sent, not the dataset itself: zfs snap storage/test-source@one;
3. Now, let’s send this snapshot to a destination dataset: zfs send -v storage/test-source@one | zfs recv -v storage/test-destination; we now have a working independant copy of the dataset on the destination, ain’t this cool?
4. Say we did some changes, and we wanna back it up! Let’s take a second snapshot: zfs snap storage/test-source@two;
5. Now, let’s send the increment to the same target: zfs send -v -i storage/test-source@two | zfs recv -v storage/test-destination
6. And… voilà ! One can list snapshots via zfs list -t snap to check what’s up.

Note: To track the progress of zfs send | zfs recv, one can use a well-known tool, pv, as suggested in the Solaris documentation.

ZFS: {zpool,dataset} tuning

Definitions

Most of us came from traditional storage systems, and had to wrap our minds around new concepts introduced by zfs. Let’s break it down:

zpool (RAID array + volume manager)

A zpool combines multiple physical disks into a single storage pool, handling redundancy, caching, and data integrity at the block level. It’s comparable to a:

• RAID array, but more flexible and self-healing;
• volume manager, dynamically allocating storage without requiring fixed partitions;
• storage backend, on top of which ZFS datasets (filesystems) are created;

Instead of manually partitioning disks or setting up traditional RAID, zfs automatically distributes data across the zpool.

ZFS: pool layout

Goals

Using ZFS for NAS appliances storing large media files is a very common scenario. In this specific context, we may want to get:

1. great storage capacity;
2. good resilience;
3. decent performance;

…in that order!

We’ll try & figure out proper storage designs for 8 large hard disks in this specific context.

Concepts

You may well want to read choosing the right ZFS pool layout in addition to getting familiar with ZFS concepts.