Skip to content

Back up and restore otomo

otomo keeps everything players and staff have ever created in one Postgres database server, and everything that makes that data usable (signing keys, passwords) in a folder of secrets. Lose either and the game is gone. A backup captures both, encrypted, every night, and a restore drill proves you can get them back. This guide covers setting up backups, checking they ran, and practising a restore. It takes about 30 minutes the first time.

You'll need:

  • SSH access to the host you are backing up (see Before you begin);
  • a trusted computer (not the server) to hold the private key and run the drill;
  • a copy of the otomo repository on that computer, and a Postgres client (psql, pg_restore) plus age.

What a backup contains

One file per run, otomo-backup-<time>.tar.gz.age, with a .sha256 file beside it:

Inside Why it matters
A dump of every database (auth, session, economy, config, patch, …) The players, inventories, matches and published content
deploy/secrets The keys that sign logins. Lose them and every player is signed out and every device binding breaks
deploy/.env Every password the stack runs with
A manifest Which git commit, profile and image tags the backup was taken from

The whole file is encrypted to a public key that lives on the server. The server can make backups but cannot read them. Only the private key, which you keep elsewhere, can. That is the point: a stolen server or a leaked copy of a backup reveals nothing.

Step 1: Make the key pair (once, on your own computer)

deploy/scripts/backup-keygen.sh --tool age --out ./otomo-backup-key

This writes two files:

  • backup.age.pub: the public key. Not secret.
  • backup.age.key: the private key. Whoever has it can read every backup and every secret inside it.

Put the private key in your team's password manager and keep a second copy somewhere physically different (a USB stick, or paper in a drawer). Then delete the copy on your computer. Never put it on the NAS, the VM, in the repository, or in chat. A backup whose only key was on the server that burned down is not a backup.

Lose the private key and the backups are unreadable

There is no recovery. Check the stored copy works (Step 4) before you trust it.

Step 2: Put the public key on each host

On the host (the school VM for staging; the otomo-prod container on the NAS for production):

sudo apt install age
cp backup.age.pub ~/otomo/deploy/secrets/backup.age.pub    # on production: /root/otomo/deploy/secrets/

Write the host's settings, then install and start the nightly timer:

echo 'OTOMO_BACKUP_OFFSITE=none' | sudo tee /etc/otomo-backup.conf   # see "Off-host copies" below
sudo chmod 600 /etc/otomo-backup.conf
sudo cp deploy/systemd/otomo-backup.service deploy/systemd/otomo-backup.timer \
        deploy/systemd/otomo-backup-failed@.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now otomo-backup.timer

The unit files assume a user and a path

The units in the repository run as team45 from /home/team45/otomo, which is the VM. On the NAS the checkout is /root/otomo and the user is root, so change User= and the three paths in the copies you install.

The timer runs every night between 03:00 and 03:45, and after a reboot if a run was missed.

Step 3: Run one now and look at it

sudo systemctl start otomo-backup.service
journalctl -u otomo-backup.service -n 30
ls -l ~/otomo-backups/daily

You should see ok next to each database, then a file ending in .tar.gz.age with a plausible size (hundreds of kilobytes to a few megabytes at this scale). Archives are kept for 7 days and 4 weeks. The log ends with a warning while off-host copies are switched off. That is deliberate.

Step 4: Practise a restore

Do this on your own computer, so the private key never goes near a server.

  1. Copy the newest archive and its .sha256 file from the host to a scratch folder.
  2. Start a throwaway Postgres (a local install, or docker run postgres:17) and point the tools at it: export PGHOST=127.0.0.1 PGPORT=5432 PGUSER=postgres OTOMO_PROFILE=staging.
  3. Check the archive without changing anything:
deploy/scripts/restore.sh otomo-backup-<time>.tar.gz.age --target scratch \
    --key ./backup.age.key --dry-run

It decrypts, checks every checksum, and lists the databases it would restore. 4. Restore for real into copies named restore_<database>:

deploy/scripts/restore.sh otomo-backup-<time>.tar.gz.age --target scratch \
    --key ./backup.age.key
  1. Read the verification. The script counts rows in the core tables and picks the newest player, login session and match, then checks they hang together. Every line should say PASS. A brand-new staging environment has no finished matches, so the match check says FAIL: no rows; re-run verify-restore.sh --prefix restore_ --allow-empty and read that as "nothing to check", not as a pass for matches.
  2. Drop the restore_* databases and delete the archive copy.

A restore into scratch takes about 10 seconds at today's data size. Record each drill in the restore runbook's drill log.

The drill never touches a live database

--target scratch only creates restore_* databases next to the real ones. Restoring over a live environment is a different command and needs extra flags; production refuses it unless you pass --i-know-this-is-production.

Off-host copies

A backup on the same disk as the data protects against mistakes, not against losing the disk. Set OTOMO_BACKUP_OFFSITE in /etc/otomo-backup.conf to one of:

Value Where it goes
rclone:<remote>:<path> Object storage (Backblaze B2, S3, R2). Best for the school VM
rsync:user@host:<path> Another machine over SSH
dir:<path> A different disk or dataset mounted on the host. For the NAS, use another pool
none Nothing leaves the host. Allowed, but every run warns and an alert eventually fires

Each upload is read back and compared by checksum before the run counts as a success.

When something is wrong

You see It means Do this
otomo-backup FAILED: in journalctl -t otomo-backup -p crit The run stopped; the old archives are still there Read the line above it; fix; start the service again
OTOMO_PROFILE is not set The host has not been told whether it is staging or production sudo deploy/scripts/set-profile.sh staging (or production)
No .tar.gz.age after a run No public key on the host, so the run refused to write a plain archive Step 2
SHA256SUMS check fails in a restore The archive was damaged in transit Copy it again; if it still fails, use the previous night's
no identity matched Wrong private key for this archive Use the key whose public half was on the host that night

Backups also write otomo_backup.prom for Prometheus. The alerts are BackupStale (no success for 26 hours), BackupFailed and BackupNoOffsiteCopy.

  • Deployment and operations: how backups fit the rest of the stack.
  • deploy/docs/BACKUP.md and deploy/docs/RESTORE-RUNBOOK.md in the repository: every flag, the encryption options, and the drill log.