usc/cookbook/zfs-send/

Usage

Basics

The following example aims at illustrating the core concept of zfs send/recv:

Let’s create a dataset via zfs create storage/test-source;
Next, we’ll take a snapshot, since only snapshots of a live filesystem can be sent, not the dataset itself: zfs snap storage/test-source@one;
Now, let’s send this snapshot to a destination dataset: zfs send -v storage/test-source@one | zfs recv -v storage/test-destination; we now have a working independant copy of the dataset on the destination, ain’t this cool?
Say we did some changes, and we wanna back it up! Let’s take a second snapshot: zfs snap storage/test-source@two;
Now, let’s send the increment to the same target: zfs send -v -i storage/test-source@two | zfs recv -v storage/test-destination
And… voilà ! One can list snapshots via zfs list -t snap to check what’s up.

Note: To track the progress of zfs send | zfs recv, one can use a well-known tool, pv, as suggested in the Solaris documentation.

Reference: see Sending and Receiving ZFS Data from the Oracle Solaris ZFS Administration Guide.

Recursive

One might be tempted to take recursive snapshots, in order to snap all of the dataset’s children in one command: zfs snap -r dataset.

This is cool, but things can quickly get out of hand with a little automation, resulting in… a very large number of snapshots. So, as a reminder, here’s a magic command to get rid of them in case of need 😅 :

zfs list -H -o name -t snap -r dataset | \
    xargs -n1 zfs destroy -nv

Note: use dry-run mode (zfs destroy -n) initially, to make sure you’re on target!

After performing recursive snaps, one might be tempted to use zfs send -R, to send them all in one go. Sure! However, we’d better understand its implications, especially when dealing with… natively encrypted datasets 😬

Encrypted datasets

Natively encrypted datasets are great, and have become quite common an option for selectively securing data at rest. However, they can also lead to complications, as reflected in this exchange on #zfs:

17:09   iio7| I have a dataset that is encrypted with ZFS native
              encryption, which I would like to send to another box
              without ZFS native encryption. The receiving box runs on
              some disks that are encrypted using LUKS, and I would
              like the receiving side to not use ZFS native
              encryption. Is it possible to send an encrypted dataset
              unencrypted?
17:14 Tabmow| It will do that by default I believe, unless you specify
              --raw (man 8 zfs-send <-- See -w / --raw)
17:17   iio7| I tried that, but when I do that, I get "cannot send",
              "encrypted dataset may not be sent with properties
              without the raw flag".
17:18 Tabmow| What command are you using? And are your keys loaded?
17:18   iio7| Yes, the key is loaded, I have updated some files.
17:20   iio7| zfs send -R vault/tank@snap | ssh foo \
                  "doas zfs receive pool2/bar"
17:20 Tabmow| Read the documentation on -R
17:21   iio7| I have. What am I missing?
17:21 Tabmow| Did you read the last line? If the -R flag is used to send
              encrypted datasets, then -w must also be specified. So I
              guess you can send unencrypted without -R - or encrypted
              with -R.
17:23   iio7| Hmm. It seems so. Just tested that. Tabmow, thanks.
17:23 Tabmow| You're welcome.
04:17    PMT| The specific problem is -R implies -p, which requires -w
              on encrypted datasets.

Using zfs send -R with encrypted datasets is… problematic, to say the least. A bug report has been opened since June 2020, with no resolution in sight. Makes you want to test it, right? 🤓

Testing

Recursive snapshots

Let’s take a recursive snapshot, in order to include all of the dataset’s (non-encrypted) children:

zfs list -r storage/unencrypted/test
NAME                           USED  AVAIL  REFER  MOUNTPOINT
storage/unencrypted/test       682K  20.0T   288K  /srv/unencrypted/test
storage/unencrypted/test/one   394K  20.0T   128K  /srv/unencrypted/test/one
storage/unencrypted/test/two   128K  20.0T   128K  /srv/unencrypted/test/two
storage/unencrypted/test/three 139K  20.0T   139K  /srv/unencrypted/test/three

zfs snap -r storage/unencrypted/test@snap

Recursive & replicative send to an encrypted dataset

Let’s send these snaps recursively, using the --replicate option of zfs send to a target dataset, that has the encryption property enabled:

zfs get encryption storage/encrypted
NAME              PROPERTY    VALUE        SOURCE
storage/encrypted encryption  aes-256-gcm  -

zfs send -R storage/unencrypted/test@snap | \
zfs recv storage/encrypted/test

Now, let’s check the results:

zfs get -r encryption storage/encrypted/test
NAME                                PROPERTY    VALUE        SOURCE
storage/encrypted/test              encryption  aes-256-gcm  -
storage/encrypted/test/one          encryption  off          default
...

Because of -R, the target datasets did not inherit the encryption properties of the parent dataset! Definitely something to keep in mind…

Non-recursive & non-replicative send to an encrypted dataset

Let’s clean this up with zfs destroy -r storage/encrypted/test, then try sending a snapshot without --replicate:

zfs send storage/unencrypted/test@snap | \
zfs recv storage/encrypted/test

This time, the received dataset has inherited the encryption property of the parent dataset indeed!

Conclusions

Let’s sum it up:

Sending encrypted data to an encrypted dataset: the only way to inherit properties is to send an unlocked dataset via zfs send without -R & -w (no recursion possible).
Sending unencrypted data to an encrypted dataset: similarly, zfs send must be used without -R.
For both cases: Ensure the target’s parent dataset is encrypted.
Sending unencrypted data to an unencrypted dataset: maximum flexibility with zfs send options.

ZFS permissions

After initial testing, we’ll rely on regular (non-root) users to perform zfs send & zfs recv operations. In order to do this, they must be granted appropriate permissions. This is where zfs allow comes into play, allowing very granular control over what one can and cannot do with zfs!

ZFS send

The following necessary permissions are requireed for using zfs send:

send: allows the user to send snapshots;
snapshot: allows the user to create snapshots, which may be necessary if new snapshots need to be created during the send operation;
hold: allows the user to create and destroy holds on snapshots, which can be useful to prevent snapshots from being deleted during the send operation;
mount: allows the user to mount datasets, which may be necessary to access data during the send operation;
traverse: allows the user to traverse the dataset hierarchy, which is required for the -R option;

In practice:

zfs allow -u $user \
    send,snapshot,hold,mount,traverse \
    $path/$to/$dataset

ZFS receive

Here’s a minimum amount of permissions required for a user to use zfs recv (one may need more for special usecases):

receive: allows the user to receive ZFS send streams;
create: allows the user to create new datasets, which may be necessary if the received stream includes new datasets;
mount: allows the user to mount datasets, which may be necessary to access data after it has been received;