This is a step-by-step guide to set up an Ubuntu server with the root partition on btrfs raid1, encrypted with luks, and automatically unlocking on boot via a Mandos server running locally.

I wanted to do this on a home server because raid and encryption makes me sleep better, but neither do I want to have to type the LUKS password every time I need to reboot the machine. There are guides out there for each part of this, but not together, which unfortunately, made setting this up more of a pain than I anticipated. Hopefully, after reading this, it should be a breeze for others.

What we want and prerequisites

We have a server with two disks in it (or two SSD-s) that are the same size (/dev/sda and /dev/sdb later on) and UEFI boot. We want to encrypt both disks, put a btrfs raid1 filesystem on them and then install an Ubuntu server on this. We also want to be able to reboot this server without needing to type our password anywhere, so we will install a mandos client on it. We will also need another machine that will act as our mandos server (I used a rasberry running Ubuntu for this). We will also need an Ubuntu Live USB (for the record, I used one with jammy).

Theory

I’m by no means an expert, and what I’m writing here might be wrong in many ways, but this was in my head by the time I finished, and thinking about it this way helped me troubleshoot.

btrfs raid1 with encryption

Btrfs doesn’t know how to do this on it’s own. So we need to encrypt a partition on each disk with luks first, unlock the encryption on both and then use btrfs to install a raid1 on the unlocked partitions. The raid1 “partition” will receive a UUID and we need to mount root (/) on this UUID and not either of the underlying partitions. Also, the Ubuntu installer doesn’t know how to install itself on this raid1, which will bring some fun.

boot order and partitioning

First thing to turn on is the BIOS, which looks for a FAT32 partition marked as EFI. In our setup this partition needs to have grub installed on it, which will take over from the BIOS. This EFI partition doesn’t have anything to do really with how we mount things on Linux, but often it is mounted under /boot/efi and when we update-grub this is where we are putting files. The actual efi stuff is placed in a folder called EFI (so when this FAT32 partitions is mounted under /boot/efi the efi stuff will be in /boot/efi/EFI). In my experience if the partition marked as EFI has other things in it or EFI is in a subfolder on the partition BIOS still finds it. On the other hand, do-release-upgrade in Ubuntu will complain for some reason, so the best is to stick to the above.

Next the initial ram filesystem is loaded into memory (the initramfs). This actually feels like a mini-linux, with a busybox shell if you ever happen to get stuck here. This is also what can bring up initial networking and where we can install the mandos client. Since we need to contact the mandos server via the client before we can unlock the encrypted disks, the initramfs images cannot be encrypted. This is what is mounted under /boot. Technically, /boot and /boot/efi could all be on the same FAT32 partition, but as I mentioned do-release-upgrade does not like this for some reason, and also FAT32 can’t do symlinks and update-initramfs by default makes some, so it’s better to create a separate ext4 or similars partition for /boot.

Setup

Boot the machine from the live USB. I will assume the other machine that will be the mandos server is already up and running on the local network with an Ubuntu server on it.

Partitioning

Since we want to do raid1, both disks should be partitioned the same way (using gparted for this should be the easiest):

/dev/sda1: 512 MB FAT32
/dev/sda2: 2 GB ext4
/dev/sda3: optional swap partition
/dev/sda4: leave the rest for encryption and raid

Repeat for /dev/sdb.

The first 3 partitions will NOT go into raid and the boot process will only be reading from one of the disks and two separate swap partitions will be available to mount. Everthing under /boot can easily be regenerate, but rsync could be used to keep the other disk in sync, making a switch easier if it would be needed. The swap can also be encrypted, but I didn’t bother (yet) and there should be ample guides for doing that anyway.

Encryption and raid

We create two encrypted partitions. Use the same password for both. Also, if something goes wrong, you’ll be unlocking the disks in initramfs which loads an English keyboard layout, so make sure you know how to type it if your usual layout is different:

cryptsetup luksFormat --type=luks1 /dev/sda4
cryptsetup luksFormat --type=luks1 /dev/sdb4

Let’s unlock the encrypted disks and name them:

cryptsetup open /dev/sda4 cryptroot1
cryptsetup open /dev/sdb4 cryptroot2

Finally, we create the btrfs raid:

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/cryptroot1 /dev/mapper/cryptroot2 -L btrfsroot

Note the UUID for the raid, but you can always get it with sudo btrfs filesystem show:

❯ sudo btrfs filesystem show
Label: 'btrfsroot'  uuid: a490e9ca-2ceb-48eb-8656-e4c311495ace
        Total devices 2 FS bytes used 93.09GiB
        devid    1 size 229.97GiB used 96.01GiB path /dev/mapper/cryptroot1
        devid    2 size 229.97GiB used 96.01GiB path /dev/mapper/cryptroot2

Installing Ubuntu

Since the live ubuntu’s installer won’t be able to do this for us, we’ll be doing the install with debootstrap and chroot.

First, let’s do some mounting. We will reproduce the partition mountings we discussed above under /mnt and we also need to mount a special efivarfs that seems to explain to linux how to configure the EFI partition properly.

(Note, I’m writing this from notes and memory, you may or may not need sudo here):

# change the UUID to what your btrfs raid is
sudo mount UUID=a490e9ca-2ceb-48eb-8656-e4c311495ace /mnt

sudo mount /dev/sda2 /mnt/boot
sudo mount /dev/sda1 /mnt/boot/efi
sudo mount -t efivarfs efivarfs /mnt/sys/firmware/efi/efivars

Now we install a very minimal Ubuntu. Unfortunately, there was no noble in deboostrap for me, which might have been because of the older Ubuntu:

sudo debootstrap jammy /mnt

Setting up Ubuntu

Next we do some more mounting, so we can chroot into our new install:

for dir in /dev /proc /sys /run; do sudo mount --bind $dir /mnt$dir; done
sudo chroot /mnt

Once in, we are going to finish the setup:

echo "mandosclient" > /etc/hostname
locale-gen en_US.UTF-8

apt update
apt install linux-image-generic linux-headers-generic grub-efi-amd64 btrfs-progs cryptsetup lvm2 initramfs-tools systemd-sysv network-manager keyutils

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
update-grub
apt install ubuntu-server # debootstrap install a very minimal system

We also make sure that our crypt setup sticks, so we edit /etc/crypttab:

# <target name> <source device>         <key file>      <options>
cryptroot1 UUID=e89263d6-aa19-4f45-b16c-5f08186aa70d none luks,discard,keyscript=/usr/lib/cryptsetup/scripts/decrypt_keyctl
cryptroot2 UUID=94281c39-5172-41fc-92e0-ec7bc47b9b83 none luks,discard,keyscript=/usr/lib/cryptsetup/scripts/decrypt_keyctl

The above UUID-s should be the UUID of your /dev/sda4 and /dev/sdb4 respectively (use blkid to get them if needed). The keyscript we added caches the first decrypt password entered during boot and tries to apply to all other encrypted partitions. This is useful already, but with mandos it is absolutely necessary since mandos will only provide a single password, so the second disk can only be unlocked if we cache the password (which is also why we needed to use the same password for both disks).

As a reference this is how /boot looked for me at this point:

❯ ls /boot
System.map-5.15.0-140-generic  config-5.15.0-140-generic  efi   initrd.img-5.15.0-140-generic  vmlinuz-5.15.0-140-generic
System.map-6.8.0-60-generic    config-6.8.0-60-generic    grub  initrd.img-6.8.0-60-generic    vmlinuz-6.8.0-60-generic

❯ tree /boot/efi
/boot/efi
└── EFI
    ├── BOOT
    │   └── BOOTX64.EFI
    └── ubuntu
        ├── grub.cfg
        └── grubx64.efi

If we rebooted now, we would be prompted once for the luks password, and we should be booted into a functioning Ubuntu on encrypted btrfs raid. The next part will require us to produce a configuration file on this machine which will include a long encryption key and transfer it to the machine that will be the mandos server. This might be easier after a reboot, but I guess you can install sshd on the live Ubuntu or ssh from the live Ubuntu to the mandos server.

Set up mandos client

Install the mandos client:

sudo apt install mandos-client

Unfortunately, there’s a bug in the package, so we need to edit /usr/share/initramfs-tools/hooks/mandos. Find the line

libgpgme11_version="`dpkg-query --showformat='${Version}' --show libgpgme11`"

and add the star at the end like so:

libgpgme11_version="`dpkg-query --showformat='${Version}' --show libgpgme11*`"

otherwise, you will see this error on booting:

Mandos plugin mandos-client: bad_gpgme_op_decrypt gnupg no secret key

We now generate the config, and copy the output to the mandos-server (put it in a file somewhere for the time being).

sudo mandos-keygen --password --type RSA --force

Finally, we regenerate the initramfs:

sudo update-initramfs -u -k all

Configure the mandos server

Go over to the server that will act as the mandos-server and install mandos.

sudo apt install mandos

Now put the output we generated on the client at the end of /etc/mandos/clients.conf.

It should look something like this (I deleted most of the values):

[mandosclient]
host = mandosclient
key_id =
fingerprint =
secret =
checker = ssh-keyscan -t ecdsa-sha2-nistp256 %%(host)s 2>/dev/null | grep --fixed-strings --line-regexp --quiet --regexp=%%(host)s" %(ssh_fingerprint)s"
ssh_fingerprint =

There’s only one thing you need to change: I found that the host was not found reliably this way, which is a problem for the checker (if that fails for long enough the server will refuse to give out the password), so I changed that value to the actuall IP of the client machine. Also note that although you can set up mandos differently, I’m assuming server and client are on the same local network.

After the setup, we enable all clients:

sudo mandos-ctl --enable --all

Then switch back to the client machine and test it:

/usr/lib/x86_64-linux-gnu/mandos/plugins.d/mandos-client --pubkey=/etc/keys/mandos/pubkey.txt --seckey=/etc/keys/mandos/seckey.txt --debug --tls-privkey=/etc/keys/mandos/tls-privkey.pem --tls-pubkey=/etc/keys/mandos/tls-pubkey.pem; echo

If things looks happy, reboot and rejoice as your server is now being unlocked automatically. Word of warning: although it’s possible to set it up, the initramfs by default does NOT have wifi, so the client machine should be hooked up to ethernet.

Set up dropbear for backup

dropbear is a small ssh server which can be installed in the initramfs and connected to over ssh to unlock the encrypted partitions over ssh should mandos fail:

sudo apt install dropbear-initramfs
sudo vim /etc/dropbear/initramfs/dropbear.conf # check the settings
sudo cp ~/.ssh/authorized_keys /etc/dropbear/initramfs/ # these keys will work now during boot as well
sudo update-initramfs -u -k all

Troubleshooting

If you are dropped in initramfs, just unlock the encrypted partitions, exit and boot should continue:

cryptsetup --verbose luksOpen /dev/sda4 /cryptroot1
cryptsetup --verbose luksOpen /dev/sdb4 /cryptroot2
exit

Some other notes

I needed reinstall keyutils package at some point, when the keyscript wasn’t working
there’s no wifi during boot
you can always chroot from the live USB again to fix things

Secure and redundant server setup with a bit of lazyness