In addition to basic SSH setup, this document will also review some more advanced SSH features like ProxyJump, ControlMaster, and some other overlooked features. Although not strictly required, since Zelta needs to communicate frequently with partner hosts, these features can be very helpful for small setups and indispensable at scale.
Zelta is designed to work well with privilege separation and without root access, and as will all aspects of systems administration, we recommend dropping privileges whenever it's convenient. But some circumstances are appropriate for running replication jobs as root, which is quicker and more convenient to set up.
On your Zelta host, as root, make an SSH key pair if you haven't already:
ssh-keygen
Since Zelta makes many connections to the remote servers, it will be convent to use ssh-agent or leave the key password blank (and obviously, make sure no one has access to your public key file). Consult the ssh-keygen and ssh-agent manages for more detail. Make a note of the created public key file, e.g. cat ~/.ssh/id_rsa.pub. We'll need the contents of this file copied into the /.ssh/authorized_keys file on remote services we intend to replicate with.
root:Inside /etc/ssh/sshd_config, uncomment and change the line that says:
PermitRootLogin prohibit-password
And reload your sshd process.
Finally, add contents of the public key file from the first step to root's ~/.ssh/authorized_keys file:
mkdir ~/.ssh
chown 700 ~/.ssh
And then paste the contents of the above public keys file into ~/.ssh/authorized_keys.
When you can ssh into your remote servers, you're ready to optimize SSH further.
Follow the same advice as above for non-root users, but skip step 2.
To-do: Provide a link to cipher benchmarks.
Example addition to ~/.ssh/config:
Host *
Ciphers aes128-ctr
KexAlgorithms ecdh-sha2-nistp256,curve25519-sha256@libssh.org
Macs umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com
Compression no
Unless your datasets are uncompressed, which is unlikely, there is very little to gain (and a lot of CPU power to waste) by enabling SSH compression for use with ZFS replication.
As an alternative to a VPN, you can use a public IP bastion as an intermediate SSH host. Our benchmarks revealed better performance using ProxyJump over all tested VPNs including WireGuard, so we exclusively use SSH jump hosts for SSH replication. To use the ProxyJump create a domain section in your ~.ssh/config specifying your jump host, like so:
Host *.chi0
ProxyJump fw2.chi0.belltower.it
You can configure a jump host for an individual host with additional overrides:
Host ca.bts
Hostname 10.2.19.77
User backup
ProxyJump fw1.fmt3.belltower.it
By configuring the final example, ssh ca.bts will automatically jump off of the indicated firewall, connect to the indicated IP, and log in as user backup.
Zelta works, like many utilities, by using a repeated series of ssh commands to "look before it leaps." Thankfully, SSH has the ability to maintain connections, and it only needs to be configured on the system running Zelta.
mkdir ~/.ssh/sockets/;chmod 700 ~/.ssh/sockets/
And then in your ~/.ssh/config file, add the following lines to the Host * section (or your required non-global sections):
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 5m
If you'd like to limit your ControlPersist time, consider experimenting to find a value that works well with your Zelta policy.
On your backup servers, create a backup user, which we will call "backupuser" user and give it access to replicate to your backup target, e.g.,
zfs allow -u backupuser receive,mount,create,readonly backuppool. Note that backupuser will not be able to receive any property it doesn't have permission to. Missing some permissions doesn't prevent a successful backup of data, but if you'd like to back up a default FreeBSD system with all properties, set these additional permissions: canmount,mountpoint,setuid,exec,atime,compression.
On your source servers (the servers with datasets you would like to back up) also create a backup user, and grant it access to send snapshots from your source dataset (or a parent dataset above your backup source dataset), e.g., zfs allow -u backupuser send,snapshot,hold pool/stuff.
Use SSH (key-based authentication)[https://docs.freebsd.org/en/books/handbook/security/#security-ssh-keygen] between your backup server and source servers. Make sure you can ssh from the machine you're running Zelta on to the others.