Skip to content

Config, Patch and content delivery

Two services deliver everything that is "data" in a live game: Config is where staff write it, and Patch is how players' games and otomo's own services read it. This page follows a change from a designer's draft to a running game, and shows how the pieces are split so that a broken Patch can never alter a release.

Content and patching explains the same vocabulary for a game developer, and the SDK side of it. Here we describe what the servers do.

The vocabulary, as the database sees it

Term What it is Mutable?
Namespace A named JSON document, such as balance.player. Created once with an audience: client (sent to players' games) or server (read only by otomo's services and game servers) The audience never changes
Schema A JSON Schema attached to a namespace. Each replacement gets a new schema version Replaced, never edited in place
Draft The working copy of a namespace. Saved with a revision number so two editors cannot silently overwrite each other (409 on a clash) Yes
Version A numbered, frozen copy of a draft (v1, v2...). Validated against the schema when created No
Pack An uploaded Godot .pck file, named and stored by SHA-256 No
Release A numbered, immutable set: one version of each chosen namespace, a list of packs, a minimum client version, and optionally a client_update block No
Channel dev, staging or live. Each has a head: the one release it is serving The head moves
Blob A stored document or pack, content-addressed (named by its SHA-256) No

Everything that matters is immutable. The only mutable things are drafts and the three channel heads. Publishing sets a head. Rollback points a head at an earlier release; promote copies one channel's current release to the next as a new release.

The authoring flow

 namespace ──edit──▶ draft ──validate──▶ version ──┐
                                                   ├──▶ release ──publish──▶ channel head
 .pck file ──upload──────────────────▶ pack ───────┘     (dev | staging | live)

Who may do each step is set by role (see Identity and sign-in):

Step Minimum role
Read anything viewer
Save and validate a draft, create a version, upload a pack live_ops
Publish to dev or staging live_ops
Create a namespace, replace a schema admin
Publish to live, roll back, promote admin

Two details are worth knowing:

  • The live publish is a separate route. POST /api/admin/config/channels/live/releases is registered as a literal path next to the {ch} wildcard, so "only an admin may publish to live" is expressed in the route table rather than in a handler. gateway_dev has the same entry, so the bar holds at both layers.
  • A publish is one transaction. Config locks the channel's head row, resolves the chosen versions, writes each document to blob storage, builds both manifests, inserts the release, moves the head, writes an audit row, and sends a NOTIFY config_release to Postgres. Either all of it happens or none of it does.

Config ships a set of starter namespaces with config seed (existing ones are skipped), for example balance.player, features.flags, ui.motd, session.rules, session.loadout, session.rank, matchmaking.rules, chat.rules, economy.rules, patch.curation and patch.curated. Namespaces such as economy.shop, economy.gacha, economy.tasks and announcements are created by an admin; see Namespace presets for a worked example.

Two manifests per release

A release produces two documents, both hashed over a canonical serialisation (sorted keys, no extra whitespace) so the same content always has the same hash:

Manifest Contains Served to By
Client manifest Client-audience namespaces and the packs, each with size and SHA-256, plus min_client_version Players' games Patch, on the public listener through the player gateway
Server manifest Server-audience namespaces only Otomo services and game servers Patch on its internal listener (:8081), and Config's bundle route

A server-audience document never appears in the client manifest, and there is no request parameter that could ask for one: this is what makes session.rules or economy.rules safe to keep out of players' hands.

Patch: read-only by design

Patch holds no authoring logic. It reads Config's database as a read-only Postgres role (patch_ro) and mounts Config's blob volume read-only, so a compromised Patch container cannot change a release.

How it stays current:

  1. On start, Patch loads every channel's head into an immutable in-memory set and checks each manifest's hash. A channel whose release fails the check keeps its last good release.
  2. It then LISTENs for config_release. A publish or rollback is picked up within moments.
  3. As a fallback it also polls every 60 seconds (PATCH_POLL_INTERVAL), and does a full reload after any lost connection.

Serving is cheap because the data is in memory and releases never change:

Route Audience Notes
GET /patch/v1/live/manifest Anyone The client manifest of the live head. Strong ETag; If-None-Match gets 304 with no body. X-Min-Client-Version header
GET /patch/v1/dev/manifest, /staging/manifest Any staff token Unshipped builds, so they need a staff token, but they are routed through the player gateway
GET /patch/v1/{channel}/client Same rules The small launcher document: release id, minimum version and the update-gate block
GET /patch/v1/blob/{sha256} Anyone Streams a document or pack. The hash is validated before the filesystem is touched; Range is supported; cache headers say immutable
GET /internal/patch/server-manifest/{channel} Service key Session, Matchmaker and Chat read their rules here
GET /internal/patch/client/{channel} Ops key The update policy for Ops, with a long-poll (?wait=) so a release reaches Ops quickly

Recorded content

Patch also owns one thing it can write: recordings, such as match replays that a game server uploads at run time. Their index lives in Patch's own database and their bytes on Patch's own volume, never in Config's. Players list the recordings they were in (or a tag, or a curated list) and download them; a game server uploads and deletes them with its service key. In a peer-hosted match the host player uploads its own replay with a player token, after Patch has checked with the Allocator that it hosted the match and with the Matchmaker who was in it; the replay stays private until staff approve it, players can report one, and a sweep deletes old ones (see Record and share replays). Recordings are never part of a release. Staff define rule-driven lists in the patch.curation namespace; see services/patch/README.md.

How clients and servers pull

All pulling is a conditional GET: ask with the ETag you already hold, and a 304 costs almost nothing.

Who What it pulls When Auth
Godot client (the SDK) Client manifest, then missing blobs At start-up (StartAsync). Interrupted downloads resume; each blob is hash-checked None for live
Session, Matchmaker, Chat Server manifest, for session.rules, matchmaking.rules, chat.rules and others Poll with If-None-Match (every 60 seconds by default); keep the last good document if a new one is invalid Service key to Patch
Economy The configuration bundle (economy.* documents) Refresh every 60 seconds (ECONOMY_CONFIG_REFRESH); applied whole or not at all config_service.key to Config
Game servers The configuration bundle: a gzipped tar of every server-audience namespace At start-up, then about once a minute with If-None-Match config_gameserver.key to Config
Ops The update policy per channel A long-poll per channel, plus a reconcile poll patch_ops.key to Patch

The bundle is built deterministically (fixed file order and timestamps), so the same release always produces the same bytes. The response carries ETag, Digest: sha-256=... and X-Otomo-Release. A game server should verify the digest, unpack into a new directory, swap atomically and keep the previous bundle if the new one is malformed. The 512 MiB ceiling returns 413 rather than a truncated archive. The recipe is in Run a new game server build.

Session adds one more check on the player path: if the client's X-Otomo-Release header differs from the live head, Session answers 409 release_outdated and the SDK patches again.

Rules apply without a restart

Rules documents change behaviour in the running services. A document that fails validation never replaces the last good one, and the service's metrics count the failed loads.

The update gate in one paragraph

A release can carry a client_update block: the minimum build per platform, a message in several languages, and where to send the player. Patch serves it with the manifest and on the /client document. Ops caches the live copy, and Gateway, Auth and Session use it to answer 426 client_outdated to a build that is too old. The mechanics, and how staff change it, are in Live ops. Content releases and client versions are different things: a release id says "your content is stale, patch again", a client version says "your executable is too old, download a new one".

What can go wrong

Symptom Likely cause
A publish succeeds but players do not change Patch has not reloaded: check patch_current_release_id{channel} in the dashboard, and that /readyz is 200
Patch /readyz is 503 on a new install Config has not migrated yet: Patch stays unready until it has loaded a manifest, and config migrate seeds a bootstrap release per channel
A game server keeps old rules It is serving the last good bundle because the new one was malformed or over the ceiling
Staff see a 409 saving a draft Someone else saved first; the editor offers a merge