Ian Chelig

System replication with ZFS

Using zrepl and bash, we can rebuild a system as is from incremental backups.

Published: 08/05/2026

A few months ago, I lost power during a crucial step of updating my system. This left me coming home to my system in a non-booting state. I was able to boot into a LiveCD and chroot into my system, however, upon successfully updating, and rebuilding my kernel, and initramfs, the system still wouldn’t boot. While, I am still confident this was a situation I could have rescued from, I had been wanting to make the move from btrfs to zfs for a while to try out incremental backups with zrepl. I decided to bite the bullet and reinstall my operating system, first making a backup of all of my user data.

One issue I’ve always had with my backup solutions was that in the event of an emergency, I didn’t have a full recovery solution planned. This would result in me following the exact pattern I am now. Reinstalling the operating system of my choice, and copying over useful data from backups manually. This almost always ended with a system that was only half replicated, or worse, sometimes with me missing important data entirely for months at a time. This time, I wanted to create the recovery solution as part of the process of setting up my backup solution.

Setting up zrepl

Setting up zrepl is pretty easy. My workstation is using Arch linux, so I just had to install the zrepl package, enable the service, and configure it in /etc/zrepl/zrepl.yml. I wanted to have a data-flow strategy where multiple devices (my workstation, my laptop, and my phone namely) would be able to use my home server as a central backup solution. This is known as fan-in. With zrepl, this will be accomplished by each device specifically pushing snapshots of their filesystem to the server. On the server end this is as easy as creating a job for each remote device, in nixos this looks like this:

#Backup server zrepl config

services.zrepl.enable = true;
services.zrepl.enable = true;
	services.zrepl.settings = {
	  jobs = [
	    {
	      type = "sink";
	      name = "backup_sink";
	      root_fs = ""; #Storage location on server

	      recv = {
		properties = {
		  "inherit" = [ "mountpoint" ];
		  override = {
		    canmount = "off";
		    atime = "off";
		  };
		};
	      };

	      serve = {
		type = "tcp";
		listen = ":8888";
		listen_freebind = true;

		clients = {
		  "<CLIENT IP ADDRESS>" = "<CLIENT HOSTNAME>";
		};
	      };
	    }
	  ];
	};

On the client end, I need to set up a configuration that will decide which datasets to snapshot, then connect to the server and when available, push snapshots to it. My setup takes a snapshot every 20 minutes, then, on the client, keeps the last 10 most recent snapshots. On the server we will set up a pruning policy which will keep all snapshots taken in the last hour, 24 hourly snapshots over the last day, 1 daily snapshot over the last 30 days, 12 30-day snapshots over the last year, and 1 yearly snapshot for the next 100 years (that way I’ll likely never have to think about this again). Here is what the config for this looks like:

# Client backup

global:
  logging:
    # use syslog instead of stdout because it makes journald happy
    - type: syslog
      format: human
      level: warn

jobs:
 - name: push_to_server
   type: push
   connect:
     type: tcp
     address: "<SERVER IP>:<PORT>"
   filesystems: {
     "zroot<": true,
     "<DATASET TO EXCLUDE>": false
   }
   snapshotting:
     type: periodic
     prefix: zrepl_
     interval: 20m
     timestamp_format: human
     timestamp_location: UTC
   pruning:
    keep_sender:
    - type: not_replicated
    - type: last_n
      count: 10
    - type: regex
      regex: "^manual_.*"
    keep_receiver:
    - type: grid
      grid: 1x1h(keep=all) | 24x1h | 30x1d | 12x30d | 100x365d # Pruning policy
      regex: "^zrepl_"
    - type: regex
      regex: "^manual_.*"

Once the services are running on both machine, I am able to verify that snapshots are being created and sent using zfs list -t snapshot on both machines.

Installing archiso and configuring the boot environment

With my backup solution configured, I now need a way to rescue from these backups in the event of a disaster. My plan for this is to create a custom iso of the clients linux distro (in this case, arch) that will have a bash script that will automate the process of connecting to the remote server, pulling down the data and setting the filesystem properties appropriately. I want the process to be simple. Boot into the livecd, run the script, follow the instructions, and reboot. However, in order for this to work, I need to find a way to have zrepl interact with my boot partition. This isn’t directly possible because the efi partition is formatted as FAT32 instead of as a zfs dataset. In order to work around this, I will create a zboot dataset that I will sync with my actual /boot folder, then, in the recovery script, after all the data has been synced to the target machine, I will copy the contents of this dataset onto the new systems actual /boot folder.

In order to obtain this syncing from the FAT32 boot fs on the client, to zfs so that we can take said snapshots, we will use a systemd path watching our /boot partition and a hook in our package manager to run anytime a package that touches /boot is updated. Both of these will run a script we will make that will do some simple verification of the data in /boot, then use rsync to copy the data over to the zfs partition. The script can be found here, followed by the pacman hook and the systemd path unit:

#zfsbootsync script

#!/usr/bin/env bash
set -euo pipefail

SRC="/boot"
DST="/zfsboot"

log() {
  echo "[zfsboot-sync] $*"
}

# --- Tests ---
# prevent concurrent instances
exec 9>/run/zfsboot-sync.lock
flock -n 9 || exit 0

# ensure paths exist
if [[ ! -d "$SRC/efi/EFI" ]]; then
  log "ERROR: $SRC/efi/EFI missing, refusing to sync"
  exit 1
fi

if [[ ! -d "$DST" ]]; then
  log "ERROR: $DST missing"
  exit 1
fi

# ensure destination is actually ZFS
if ! findmnt -n -o FSTYPE "$DST" | grep -q zfs; then
  log "ERROR: $DST is not a ZFS mount"
  exit 1
fi

# ensure /boot is mounted
mountpoint -q /boot/efi || {
  echo "ERROR: /boot/efi not mounted"
  exit 1
}

# ensure kernel + initramfs exist
[[ -s /boot/vmlinuz-linux-zen ]] || exit 1
[[ -s /boot/initramfs-linux-zen.img ]] || exit 1

# ensure initramfs is newer than kernel
if [[ /boot/initramfs-linux-zen.img -ot /boot/vmlinuz-linux-zen ]]; then
  echo "ERROR: initramfs older than kernel"
  exit 1
fi

# ensure microcode exists
[[ -s /boot/amd-ucode.img ]] || exit 1

# ensure rEFInd core exists
[[ -s /boot/efi/EFI/refind/refind_x64.efi ]] || exit 1
[[ -s /boot/efi/EFI/refind/refind.conf ]] || exit 1

# ensure ZFSBootMenu exists
[[ -s /boot/efi/EFI/zbm/vmlinuz-linux-zen.EFI ]] || {
  echo "ERROR: missing ZBM EFI"
  exit 1
}

# --- snapshot before overwrite ---
SNAP="zroot/data/zfsboot@pre-sync-$(date +%s)"
log "Creating snapshot $SNAP"
zfs snapshot "$SNAP"

# --- rsync ---
log "Syncing to staging area"
rsync -aHAX --delete \
--exclude='loader/random-seed' \
  "$SRC/" \
  "$DST/"

log "Sync complete"

Pacman Hook:

# /etc/pacman.d/10-zfsbootsync.hook

[Trigger]
Type = Package
Operation = Install
Operation = Upgrade
Target = zfs
Target = zfsbootmenu
Target = linux-zen
Target = linux-firmware
Target = amd-ucode
Target = refind
Target = mkinitcpio

[Action]
Description = Syncing /boot to ZFS mirror...
When = PostTransaction
Exec = /usr/local/bin/zfsbootsync

Systemd Path Unit:

#/etc/systemd/system/zfsbootsync.path
[Unit]
Description=Watch /boot for changes

[Path]
PathModified=/boot/vmlinuz-linux-zen
PathModified=/boot/initramfs-linux-zen.img
PathModified=/boot/efi/EFI
PathModified=/boot/efi/EFI/refind/refind.conf
PathModified=/boot/efi/EFI/zbm/vmlinuz-linux-zen.EFI

[Install]
WantedBy=multi-user.target

Systemd Service Unit:

#/etc/systemd/system/zfsbootsync.service
[Unit]
Description=Sync /boot to ZFS mirror

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zfsbootsync

With that done, I can move on to creating an ISO to boot from that will have the recovery script, ssh keys to connect to the remote server with, and a few other QoL changes like networking, a root password, creating a user and a sudoers file, disabling auto-login, and installing the following packages:

  • zfs
  • networkmanager
  • ssh
  • rsync
  • gptfdisk
  • parted

Since this client is on arch linux we will be using https://wiki.archlinux.org/title/Archiso. With this, all additional files I wish to be added to the ISO will be added to the airootfs folder. Once that is completed, I can create the ISO with the following command, which will copy the recovery script into the airootfs folder, create the iso, and burn it onto a usb drive using dd.

sudo cp <RECOVERY SCRIPT> airootfs/home/archie/ && sudo rm -rf isobuild && \
sudo mkarchiso -v -r -w work -o isobuild . && \
sudo dd bs=4M if=isobuild/archlinux-2026.04.16-x86_64.iso \
of=/dev/disk/by-id/<USB DRIVE> conv=fsync oflag=direct status=progress

Creating & testing the recovery script

Now that we have all the prep work done, we can get to writing the recovery script itself. This is the actual purpose of, and easily the most complex part of this project. But, once it is done, a certain level of peace of mind will be had in knowing that barring physical catastrophe, all of my unrecoverable data is protected. Here is a list I made before writing it of the process it should take:

  1. Connect to remote server
  2. Verify with user what datasets to recover
  3. Pick a snapshot time stamp
  4. Store dataset attributes
  5. Format disk and create new zpool/boot partition
  6. Use zfs send/receive to send data over
  7. Set temp mount points under /mnt
  8. Mount copied over partitions
  9. Sync /boot/efi with zfsboot partition
  10. Configure chroot environment of target
  11. Set dataset attributes
  12. Unmount partitions
  13. Export zfs pool

After a lot of trial and error testing this on my laptop, here is the script I landed on:

#!/usr/bin/env bash

set -xeo pipefail

# Config
#
DEVNAME="/dev/nvme0n1"
SRCSRV="<IP>" #Server to pull recovery from
SSHKEYFILE="/home/archie/.ssh/id_ed25519"
SSHUSER="root" #zfs send req
WIPEZFSPART="true"
RESCUEEFI="true" #WIPES WHOLE DRIVE!!!
RESCUEROOT="true"
RESCUEHOME="true"
SNAPSHOTNAME="zrepl_2026-04-18_04:14:59" #zfs list -t snapshot
  # Check for nvme drive
  if [[ "$DEVNAME" =~ nvme ]]; then
    DESTDEV="${DEVNAME}p"
  else
    DESTDEV="$DEVNAME"
  fi

validate() {
  echo "=== VALIDATION ==="


  # Does device exist?
  [[ -b "$DEVNAME" ]] || {
    echo "ERROR: bad device"
    exit 1
  }

  # Does snapshot exist?
  ssh -i $SSHKEYFILE $SSHUSER@$SRCSRV "zfs list -H <REMOTE SNAPSHOT PATH>@${SNAPSHOTNAME}" >/dev/null 2>&1 || {
    echo "SNAPSHOTNAME is empty"
    exit 1
  }

  # Print confirmation info
  lsblk "$DEVNAME"
  echo
  echo "ZFS partition wipe:  $WIPEZFSPART"
  echo "EFI restore:   $RESCUEEFI"
  echo "ROOT restore:  $RESCUEROOT"
  echo "HOME restore:  $RESCUEHOME"
  echo "Snapshot:      $SNAPSHOTNAME"
  echo "Source:        $SSHUSER@$SRCSRV"
  echo

  if [[ "$RESCUEEFI" == true ]]; then
    echo "WILL CREATE: EFI partition (512M)"
    echo "!!! THIS WILL WIPE: $DEVNAME"
    WIPEZFSPART=true
    RESCUEHOME=true
    RESCUEROOT=true
  fi

  if [[ "$WIPEZFSPART" == true ]]; then
    echo "WILL CREATE: ZFS partition (rest of disk)"
    echo "!!! THIS WILL WIPE: $DEVNAME ZFS PARTITION"
  fi

  echo
  if [[ "$WIPEZFSPART" == true && "$RESCUEEFI" == true ]]; then
    echo "!!! THIS WILL WIPE: $DEVNAME"
  fi
}

confirm() {
  # Confirm destructive task
  read -rp "Type 'WIPE' to continue: " ans
  [[ "$ans" == "WIPE" ]] || exit 1
}

destruct() {
  echo "=== DESTRUCTION ==="

  # Reformat EFI partition
  if [[ "$RESCUEEFI" == true ]]; then
    sgdisk --zap-all "$DEVNAME"
    sgdisk -n 1:0:+512M "$DEVNAME"
    sgdisk -t 1:EF00 "$DEVNAME"
  fi

  # Reformat zfs partition
  if [[ "$WIPEZFSPART" == true ]]; then
    sgdisk -n 2:0:0 "$DEVNAME"
    sgdisk -t 2:BF00 "$DEVNAME"
  fi
}

setup_fs() {
  echo "=== FS SETUP ==="

  if [[ "$RESCUEEFI" == true ]]; then
    mkfs.fat -F32 "${DESTDEV}1"
  fi

  if [[ "$WIPEZFSPART" == true ]]; then
    zpool create -f -o ashift=12 \
      -O compression=lz4 \
      -O acltype=posixacl \
      -O xattr=sa \
      -O atime=off \
      -O encryption=aes-256-gcm \
      -O keyformat=passphrase \
      -O keylocation=prompt \
      -O normalization=formD \
      -o autotrim=on \
      -m none zroot "${DESTDEV}2"
  fi
}

restore_root() {
  if [[ "$RESCUEROOT" == true || "$RESCUEEFI" == true ]]; then

    echo "RESTORE ROOT"

    # Create system dataset zroot/ROOT if doesnt exist
    if ! zfs list -H "zroot/ROOT" >/dev/null 2>&1; then
      zfs create zroot/ROOT
      zfs set mountpoint=none zroot/ROOT
      zfs set canmount=off zroot/ROOT
    fi

    # Receive dataset zroot/ROOT/arch
    if ! ssh -i "$SSHKEYFILE" \
      $SSHUSER@$SRCSRV \
      "set -euo pipefail; zfs send <REMOTE SNAPSHOT PATH>@${SNAPSHOTNAME}" |
      zfs receive -F -u zroot/ROOT/arch; then
      echo "ERROR: root restore failed"
      zfs destroy -r zroot/ROOT/arch || true
      exit 1
    fi
  fi
}

restore_home() {
  if [[ "$RESCUEHOME" == true || "$RESCUEEFI" == true ]]; then

    echo "RESTORE HOME"

    # Create system dataset zroot/data if it doesnt exist
    if ! zfs list -H "zroot/data" >/dev/null 2>&1; then
      zfs create zroot/data
      zfs set mountpoint=none zroot/data
      zfs set canmount=off zroot/data
    fi

    if ! zfs list -H "zroot/data/home" >/dev/null 2>&1; then
      zfs create zroot/data/home
      zfs set mountpoint=none zroot/data/home
      zfs set canmount=off zroot/data/home
    fi

    # Receive dataset zroot/data/home
    if ! ssh -i "$SSHKEYFILE" \
      "$SSHUSER@$SRCSRV" \
      "set -euo pipefail; zfs send <REMOTE SNAPSHOT PATH>@${SNAPSHOTNAME}" |
      zfs receive -F -u zroot/data/home/<USER>; then
      echo "ERROR: home/<USER> restore failed"
      zfs destroy -r zroot/data/home/<USER> || true
      exit 1
    fi
    zfs load-key -r zroot
  fi
}

restore_boot() {
  [[ "$RESCUEEFI" != true ]] && return
  echo "RESTORE BOOT"

  # Create system dataset zroot/data if it doesnt exist
  if ! zfs list -H "zroot/data" >/dev/null 2>&1; then
    zfs create zroot/data
    zfs set mountpoint=none zroot/data
    zfs set canmount=off zroot/data
  fi

  # Receive dataset zroot/data/zfsboot
  if ! ssh -i "$SSHKEYFILE" \
    "$SSHUSER@$SRCSRV" \
    "set -euo pipefail; zfs send <REMOTE SNAPSHOT PATH>@${SNAPSHOTNAME}" |
    zfs receive -F -u zroot/data/zfsboot; then
    echo "ERROR: boot restore failed"
    zfs destroy -r zroot/data/zfsboot || true
    exit 1
  fi

  # Load encryption key and mount drive
  zfs load-key -r zroot
  # Set mountpoint for /, /zfsboot, /boot/efi into /mnt
  zfs set mountpoint=/mnt zroot/ROOT/arch
  zfs set mountpoint=/mnt/zfsboot zroot/data/zfsboot
  zfs set canmount=on zroot/data/zfsboot
  zfs set canmount=on zroot/ROOT/arch
  mkdir -p /mnt/boot/efi
  mount ${DESTDEV}1 /mnt/boot/efi

  # Sync /zfsboot to /boot
  rsync -aHAX --delete \
    --exclude='loader/random-seed' \
    /mnt/zfsboot \
    /mnt/boot
}

chroot_config() {
  echo "CONFIGURING CHROOT"

  # Get /boot/efi partuuid
  local fstab="/mnt/etc/fstab"
  local new_uuid=$(blkid -s PARTUUID -o value "${DESTDEV}1")

  # Verify part uuid exists
  if [[ -z "$new_uuid" ]]; then
    echo "ERROR: Failed to get PARTUUID for ${DESTDEV}1" >&2
    return 1
  fi

  # Copy over new partuuid to fstab
  sed -i "\|/boot/efi| s|\(/dev/disk/by-partuuid/\)[^[:space:]]*|\1${new_uuid}|" "$fstab"


  if ! grep -q "$new_uuid" "$fstab"; then
    echo "ERROR: UUID not written correctly" >&2
    return 1
  fi

  echo "fstab updated successfully with PARTUUID=$new_uuid"
  echo "Backup created: $backup"

  arch-chroot /mnt rm /etc/zfs/zpool.cache || true
  arch-chroot /mnt rm /etc/hostid
  cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache
  arch-chroot /mnt zgenhostid
  arch-chroot /mnt zfs set org.zfsbootmenu:commandline="noresume init_on_alloc=0 rw spl.spl_hostid=$(hostid) splash" zroot/ROOT/arch
  arch-chroot /mnt refind-install
  arch-chroot /mnt mkinitcpio -P
  arch-chroot /mnt generate-zbm
}

setopts_export() {
  umount /mnt/boot/efi || true
  zfs umount zroot/data/zfsboot || true
  zfs umount zroot/ROOT/arch || true

  # Set dataset properties
  zpool set cachefile=/etc/zfs/zpool.cache zroot
  zfs set canmount=noauto zroot/ROOT/arch
  zfs set mountpoint=/ zroot/ROOT/arch
  zfs set mountpoint=/zfsboot zroot/data/zfsboot
  zfs set canmount=on zroot/data/zfsboot
  zfs set mountpoint=/home zroot/data/home
  zfs set canmount=on zroot/data/home
  zfs set mountpoint=/home/<USER> zroot/data/home/<USER>
  zfs set canmount=on zroot/data/home/<USER>

  zpool export zroot
}

main() {
  validate
  confirm
  destruct
  setup_fs
  restore_root
  restore_boot
  restore_home
  chroot_config
  setopts_export
}
main "$@"

And it works! Running this in a livecd on my laptop allowed me to reboot into a functionally identical system to my workstation!

Insights & Reflections

One of the biggest issues I had during this project was during testing. I had the idea for what the script should do, but I would be overlooking small steps, or lose internet connection halfway through and have to restart. A lot of times this resulted in me making changes to the ISO on my workstation, re-burning it onto a USB, and restarting the process of copying over my system. Could take a few hours just to get one test done. What helped me about halfway through was creating a main function and just commenting out the successful steps from previous attempts.

For larger projects like this, that deal with large quantities of data, I might have to look into test-driven development in order to make the finishing tweaks on a script like this quicker to accomplish. The final quirk I’m dealing with is that on the target machine, it asks for the encryption key twice during boot. I’m sure it will be an easy fix once I track down the cause. Moving forward I would like to fix that final issue, and find a way to resume a previously in progress rescue attempt.