How To Encrypt and Securely Backup Directories in Ubuntu

Do not let them steal your data if they steal your physical server!

Introduction

First, this story is about tools that provide cryptographically strong encryption based on the AES standard. Be warned that if you forget your password (used to derive and encrypt the AES key), you will have to wait several decades until more powerful hardware will be available. Besides, you will have to pay for that computation power to perform a brute-force search to recover your lost key.

Second, I will focus on just a few tools that I found to be the best in their class. They are fscrypt and cryfs for the on-premises encryption in case of theft and restic for encrypted remote backups.

Third, I will focus on Ubuntu since it makes things a bit easier. Possible issues can be solved by simple googling. If you don't have an Ubuntu server yet, have a look at Multipass, a cross-platform tool for creating Ubuntu virtual machines up and running inside your desktop OS in seconds. Alternatively, you can install Ubuntu on a Raspberry Pi or hire a cloud-based server.

If you don’t want to configure a server and just want to be safe on your PC or laptop, look at the Cryptomator tool. It has a user-friendly graphical interface and is available on different platforms, including non-Pro Windows 10 without BitLocker.

Full Disk Encryption vs. Directory Encyption

Full disk encryption requires a password each time a computer is started. It is a good option for PCs and laptops; you should definitely enable it if your OS supports it.

However, for servers, full disk encryption is not a convenient option: after a power outage, the server won’t boot unless you enter a password. Entering the password may be difficult if your server can be accessed only via SSH and has no monitor and keyboard attached :) Thus, I recommend per-directory encryption for servers. After a reboot, you will be able to connect to your server via SSH and mount the encrypted directory by entering the required password/key.

A pro hint. I recommend creating a script after-reboot.sh, which would ask for an AES key password and use it to decrypt the directories. Besides, the script should mount your remote backup directories.
After a reboot, connect to the server via SSH and launch the script.

Encryption Tools

There are many tools that can encrypt directories. As of early 2021, I found that Google’s fscrypt is the best option. It relies on the file system encryption code built into the Linux kernel. Thus, it is very fast and stable. The shortcomings:

• It is for Linux only; thus, it is not a good option if you wish to access your encrypted folder by attaching the physical disk to another OS.
• Currently, fscrypt supports only ext4, F2FS, and UBIFS. Since Ubuntu uses ext4 by default, this should not be a big deal.
• You have to explicitly enable the encryption (I will tell you how).
• While directory and file names are encrypted, the directory tree structure is not.
• If you wish to access files on the encrypted directory from Docker, you will need Linux v5.4 or later and fscrypt v0.2.6 or later. Thus, the default packages on Ubuntu 20.04 are NOT OK (you can build fscrpypt manually, though). Ubuntu 20.10 and 21.04 are OK.

If you find at least one of these shortcomings unacceptable, take a look at CryFS. It originated as an academic project but has already been included in the Ubuntu package list. It is cross-platform and works on top of any file system. The shortcomings:

• CryFS acts as a user-space file system driver (that’s why it is slower than kernel-based fscrypt).
• You need the latest CryFS version to deal with writes-on-read and to get the stronger encryption on disks larger than 64GB. Fortunately, it is easy to install the latest CryFS from sources by following the instructions on the CryFS GitHub page.
• CryFS can make you “cry”! The encrypted directory can be easily corrupted. It can occur when a write operation is not finished due to hardware failure or power outage. It can also occur when the raw encrypted directory is accessed by multiple concurrent processes (e.g., CryFS and some other program, or two CryFS instances, e.g., one local and one remote). Sadly, no recovery tools are available at the moment. Thus, we need to configure backups wisely in the case of CryFS (see below).

To minimize the risks, mount the encrypted CryFS directory only on a single machine at a time. Then work with the mounted directory only (which is decrypted on-the fly by the singleton CryFS instance).

Despite the shortcomings, CryFS is ahead of other similar tools (see the comparison table here).

If you are OK with the pre-packaged versions, you can install fscrypt as

sudo apt install fscrypt

and CryFS as

sudo apt install cryfs

The fscrypt Life Cycle

Initialization

First, find the name of the block device with the supported file system (e.g., ext4):

fscrypt status

Then enable the encryption for that device:

sudo tune2fs -O encrypt /dev/device

Initialize fscrypt (this operation will create the /etc/fscrypt.conf file; it is the only fscrypt operation requiring sudo):

sudo fscrypt setup /

Here “/” is the root mount point for your block device with the encryption enabled.

Now, check the status again to see that the encryption is now “supported”:

fscrypt status

Now choose a directory name and run:

fscrypt encrypt your_data_dir

The directory will be mounted (=unlocked). You can browse the files inside it as usual.

Notice that there can be problems accessing encrypted directories inside your home directory (e.g., /home/ubuntu) by other users (e.g., www-data). Ubuntu 21.04, for example, makes /home/ubuntu non-accessible to others by default.

Unmount (Lock)

To unmount (lock) the directory run

fscrypt lock your_data_dir

or, if that does not work (e.g., in case of an old fscrypt version), run (“/” is the “/” from above):

sudo fscrypt purge / --user=$USER

This will umount all fscrypt directories on the device associated with “/”.

Now, if you browse the files inside your_data_dir, you will see the encrypted file and directory names with encrypted content. The directory structure, however, will be the same as unencrypted.

Mount (Unlock)

To mount (unlock) the directory once again, run

fscrypt unlock your_data_dir

The CryFS Life Cycle

Initialization

Choose two directory names: the encrypted directory name and the mount point where you will access decrypted files (we call it your_data_dir). Then run

cryfs your_encrypted_dir your_data_dir

CryFS will ask you for a password. The AES key will be derived from your password. The encrypted config file will be stored inside your_encrypted_dir. If you wish to store the config file in a safer place, add the “--config” argument:

cryfs your_encrypted_dir your_data_dir --config my-cryfs.cfg

However, it seems that storing the encrypted password is safe enough since cryfs derives a key from the password using scrypt, which makes brute-force attacks difficult.

Unmount (Lock)

CryFS is a user-space file system driver (i.e., it works via FUSE). To unmount the mount point, just run

fusermount -u your_data_dir

Alternatively, you can invoke

cryfs-unmount -u your_data_dir

Mount (Unlock)

The full command you may wish to run is:

cryfs your_encrypted_dir your_data_dir --config my-cryfs.cfg --fuse-option noatime --fuse-option allow_other

The “noatime” option disables writes-on-read (thus, the last access time won’t be updated). The “allow_other” option is explained in the next section.

CryFS: Allow Access for Other Users

By default, FUSE allows only the owner of the encrypted directory to access it. If you want to allow access for other users, open /etc/fuse.conf and uncomment the line:

user_allow_other

Then, when mounting, specify the “allow_other” FUSE option as mentioned earlier.

Configure Automatic Encrypted Backups

First, mount your remote storage into, e.g., /mnt/remote.

Useful tools:

• To mount Windows shares, use cifs-utils.
• To mount OneDrive, Dropbox, Google Drive, and other cloud drives, use rclone.

Do not worry about Big Brother. We will encrypt our data before sending them to remote storage. We will use restic for such encrypted backups.

With CryFS, we could just sync the encrypted directory. Unfortunately, cryfs can make us“cry” if multiple processes access the encrypted directory concurrently (e.g., cryfs and restic). Thus, we will use the decrypted data from the CryFS mount point and encrypt them again with restic.

With fscrypt, we do not see the encrypted directory when it is mounted (since it was mounted in-place). Thus, the same approach applies here.

First, install restic:

sudo apt install restic

Then initialize the mounted remote directory to accept restic backups:

restic init --repo /mnt/remote

At this step, restic will ask you for a password that will be used to encrypt the AES key. Choose a reasonably strong password since the encrypted AES key will be stored in that remote directory. Like with cryfs, restic uses scrypt to derive keys from your password; thus, it is reasonably safe to keep the keys subdirectory on a remote drive.

A pro hint for panick-mongers. If you still don’t want to store the encrypted keys on a remote drive, try to create a local directory with the subdirectory “keys” and four sibling symlinks to the remote directories named “data”, “index”, “locks”, “snapshots”, as well as one symlink to the remote “config” file.

To backup your data, invoke

sudo restic -r /mnt/remote your_data_dir/subdir_to_backup

This will ask you for a key password.

If you wish to perform backups automatically (e.g., every hour using cron), create a file (say, your_data_dir/.restic) containing your restic password. I suggest putting that file into a cryfs or fscrypt mount point (but outside subdir_to_backup).

To backup your data without asking for a key password, invoke

sudo restic -r /mnt/remote your_data_dir/subdir_to_backup -p your_data_dir/.restic

If you wish to do that every hour, add this line to your /etc/crontab:

0 * * * * root restic -r /mnt/remote backup your_data_dir/subdir_to_backup -p your_data_dir/.restic

Notice that restic doesn’t follow symlinks. It syncs symlinks as symlinks.

To list the backups (called snapshots in restic), invoke

sudo restic -r /mnt/remote your_data_dir/subdir_to_backup -p your_data_dir/.restic snapshots

You can instruct restic to remove unnecessary backups according to some rules (a policy in restic terms). For instance, if we perform backup every hour, we do not need to store more than 24 hourly backups, 7 daily backups, 5 weekly, etc. To remove backups according to such a policy, invoke

sudo restic -r /mnt/remote forget --prune --keep-daily 7 --keep-hourly 24 --keep-weekly 5 --keep-monthly 6

This will do the job only once. To do it on a regular basis, add the same command to /etc/crontab (do not forget to specify the cron schedule).

☕