Back up and restore otomo¶
otomo keeps everything players and staff have ever created in one Postgres database server, and everything that makes that data usable (signing keys, passwords) in a folder of secrets. Lose either and the game is gone. A backup captures both, encrypted, every night, and a restore drill proves you can get them back. This guide covers setting up backups, checking they ran, and practising a restore. It takes about 30 minutes the first time.
You'll need:
- SSH access to the host you are backing up (see Before you begin);
- a trusted computer (not the server) to hold the private key and run the drill;
- a copy of the otomo repository on that computer, and a Postgres client (
psql,pg_restore) plus age.
What a backup contains¶
One file per run, otomo-backup-<time>.tar.gz.age, with a .sha256 file beside it:
| Inside | Why it matters |
|---|---|
A dump of every database (auth, session, economy, config, patch, …) |
The players, inventories, matches and published content |
deploy/secrets |
The keys that sign logins. Lose them and every player is signed out and every device binding breaks |
deploy/.env |
Every password the stack runs with |
| A manifest | Which git commit, profile and image tags the backup was taken from |
The whole file is encrypted to a public key that lives on the server. The server can make backups but cannot read them. Only the private key, which you keep elsewhere, can. That is the point: a stolen server or a leaked copy of a backup reveals nothing.
Step 1: Make the key pair (once, on your own computer)¶
This writes two files:
backup.age.pub: the public key. Not secret.backup.age.key: the private key. Whoever has it can read every backup and every secret inside it.
Put the private key in your team's password manager and keep a second copy somewhere physically different (a USB stick, or paper in a drawer). Then delete the copy on your computer. Never put it on the NAS, the VM, in the repository, or in chat. A backup whose only key was on the server that burned down is not a backup.
Lose the private key and the backups are unreadable
There is no recovery. Check the stored copy works (Step 4) before you trust it.
Step 2: Put the public key on each host¶
On the host (the school VM for staging; the otomo-prod container on the NAS for
production):
sudo apt install age
cp backup.age.pub ~/otomo/deploy/secrets/backup.age.pub # on production: /root/otomo/deploy/secrets/
Write the host's settings, then install and start the nightly timer:
echo 'OTOMO_BACKUP_OFFSITE=none' | sudo tee /etc/otomo-backup.conf # see "Off-host copies" below
sudo chmod 600 /etc/otomo-backup.conf
sudo cp deploy/systemd/otomo-backup.service deploy/systemd/otomo-backup.timer \
deploy/systemd/otomo-backup-failed@.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now otomo-backup.timer
The unit files assume a user and a path
The units in the repository run as team45 from /home/team45/otomo, which is the VM.
On the NAS the checkout is /root/otomo and the user is root, so change User= and
the three paths in the copies you install.
The timer runs every night between 03:00 and 03:45, and after a reboot if a run was missed.
Step 3: Run one now and look at it¶
sudo systemctl start otomo-backup.service
journalctl -u otomo-backup.service -n 30
ls -l ~/otomo-backups/daily
You should see ok next to each database, then a file ending in .tar.gz.age with a
plausible size (hundreds of kilobytes to a few megabytes at this scale). Archives are kept
for 7 days and 4 weeks. The log ends with a warning while off-host copies are switched off.
That is deliberate.
Step 4: Practise a restore¶
Do this on your own computer, so the private key never goes near a server.
- Copy the newest archive and its
.sha256file from the host to a scratch folder. - Start a throwaway Postgres (a local install, or
docker run postgres:17) and point the tools at it:export PGHOST=127.0.0.1 PGPORT=5432 PGUSER=postgres OTOMO_PROFILE=staging. - Check the archive without changing anything:
deploy/scripts/restore.sh otomo-backup-<time>.tar.gz.age --target scratch \
--key ./backup.age.key --dry-run
It decrypts, checks every checksum, and lists the databases it would restore.
4. Restore for real into copies named restore_<database>:
- Read the verification. The script counts rows in the core tables and picks the newest
player, login session and match, then checks they hang together. Every line should say
PASS. A brand-new staging environment has no finished matches, so the match check saysFAIL: no rows; re-runverify-restore.sh --prefix restore_ --allow-emptyand read that as "nothing to check", not as a pass for matches. - Drop the
restore_*databases and delete the archive copy.
A restore into scratch takes about 10 seconds at today's data size. Record each drill in the restore runbook's drill log.
The drill never touches a live database
--target scratch only creates restore_* databases next to the real ones. Restoring
over a live environment is a different command and needs extra flags; production refuses
it unless you pass --i-know-this-is-production.
Off-host copies¶
A backup on the same disk as the data protects against mistakes, not against losing the
disk. Set OTOMO_BACKUP_OFFSITE in /etc/otomo-backup.conf to one of:
| Value | Where it goes |
|---|---|
rclone:<remote>:<path> |
Object storage (Backblaze B2, S3, R2). Best for the school VM |
rsync:user@host:<path> |
Another machine over SSH |
dir:<path> |
A different disk or dataset mounted on the host. For the NAS, use another pool |
none |
Nothing leaves the host. Allowed, but every run warns and an alert eventually fires |
Each upload is read back and compared by checksum before the run counts as a success.
When something is wrong¶
| You see | It means | Do this |
|---|---|---|
otomo-backup FAILED: in journalctl -t otomo-backup -p crit |
The run stopped; the old archives are still there | Read the line above it; fix; start the service again |
OTOMO_PROFILE is not set |
The host has not been told whether it is staging or production | sudo deploy/scripts/set-profile.sh staging (or production) |
No .tar.gz.age after a run |
No public key on the host, so the run refused to write a plain archive | Step 2 |
SHA256SUMS check fails in a restore |
The archive was damaged in transit | Copy it again; if it still fails, use the previous night's |
no identity matched |
Wrong private key for this archive | Use the key whose public half was on the host that night |
Backups also write otomo_backup.prom for Prometheus. The alerts are BackupStale (no
success for 26 hours), BackupFailed and BackupNoOffsiteCopy.
Related pages¶
- Deployment and operations: how backups fit the rest of the stack.
deploy/docs/BACKUP.mdanddeploy/docs/RESTORE-RUNBOOK.mdin the repository: every flag, the encryption options, and the drill log.