Telemetry and crash reports¶
otomo can tell you two things you otherwise learn from angry players: how the game is running on their machines (frame times, hitches, load times, latency, packet loss, by game version, operating system and graphics-card maker) and what is crashing (crashes grouped together, counted, and tracked until someone fixes them). It is optional, it is off until you turn it on, and the game sends nothing until the player has agreed. This guide covers turning it on, connecting the game, reading what comes back, and what it keeps, for how long, and what it costs.
You'll need:
- a running otomo stack you can deploy to (see Deploy otomo on a VM);
- the SDK installed in your game (see Install the SDK);
- a settings screen in the game where a player can say yes or no. The decision is yours to design; otomo only insists that the default is no.
What it keeps, and what it never keeps¶
| Kept | Never kept |
|---|---|
| One summary per run of the game: version, OS family and major version, graphics-card maker, a few timing numbers, how long it lasted | The player's IP address. The telemetry service does not read it, log it or have a column for it |
| A deduplicated hardware profile (OS, CPU, GPU, memory, resolution) | The player's id. It is stored only as a one-way keyed hash (HMAC), so a copy of the database names nobody |
| Counts of the gameplay events you list (see the event list) | Free text a player typed. A text property of an event is checked and then thrown away |
| Crash reports: the error type, message, stack frames, up to 50 breadcrumbs, and optionally the last 256 KB of the game's log | Home-directory paths, e-mail addresses, IP addresses and tokens in any of that text. They are replaced before anything is stored, once by the SDK and again by the server |
Two honest limits. Scrubbing is pattern-based: it finds C:\Users\<name>, /home/<name>, e-mail
addresses, IP addresses and token-shaped strings, and it replaces more than it should rather than less,
but a player's name that appears in a log line as an ordinary word will not be found. And the gateway's
own access log still records the connecting address for every request, as it does for every other route,
for as long as your log retention keeps it. Telemetry does not add to that; it does not remove it either.
Step 1: Turn it on¶
Telemetry is a compose profile. A host that never names it runs exactly what it ran before.
- In
deploy/.env, setCOMPOSE_PROFILESto includetelemetry, for exampleCOMPOSE_PROFILES=edge,telemetry. - Run
deploy/scripts/generate-secrets.sh. It adds the service's database password to.envand writesdeploy/secrets/telemetry/hmac.key, the key the player id is hashed with. Back this key up with the rest ofdeploy/secrets(the nightly backup does). Deleting it makes everything already stored permanently unlinkable to players, which is a legitimate way to erase it and a bad way to find out. - If the script tells you to
sudo chownthe newsecrets/telemetryfolder, do that. - Run
deploy/scripts/up.sh. It creates thetelemetrydatabase and role, migrates the schema, starts the service and waits for it to report ready. - Have Prometheus scrape it. Its job is the tracked file
deploy/observability/optional/telemetry.yml. Fromdeploy/, copy it intoobservability/scrape.d/, then reload Prometheus:
cp observability/optional/telemetry.yml observability/scrape.d/
$COMPOSE up -d --no-deps prometheus # once, on a host whose prometheus predates the scrape.d mount
docker kill -s HUP otomo-prometheus-1 # afterwards a reload is enough
scrape.d/ is git-ignored, so the checkout stays clean (release.sh refuses to build from a dirty tree).
Set TELEMETRY_NEWEST_VERSION in .env to the newest game version you have published. A client reports its own
version, so one that claims a newer version than that (or one that is not shaped like a release) is counted under
"unknown" or "other" and cannot affect how crashes are grouped or reopened. Leave it empty and only the shape is checked.
To turn it off again, remove telemetry from COMPOSE_PROFILES, delete observability/scrape.d/telemetry.yml,
reload Prometheus and run up.sh. The data stays in Postgres and
in the otomo_telemetry-data volume until you delete them.
Step 2: Connect the game¶
The SDK's TelemetryClient and CrashReporter live in OtomoSdk.Telemetry. Both are off until you set
Enabled = true, and turning them off again deletes anything they queued on the player's disk before the
setter returns (a send in progress is cancelled, and nothing tracked before is sent later). The rest is
yours: build them once at start-up, set consent from your settings screen, and call them from the places
the game already knows about.
using OtomoSdk.Telemetry;
var transport = new TelemetryHttpTransport(http, config, () => auth.AccessToken, auth.RefreshAccessTokenAsync);
var telemetry = new TelemetryClient(transport, new TelemetryOptions {
QueueDir = Path.Combine(config.DataDir, "telemetry"),
SampleRate = 1.0, // 0.25 = one run in four reports
});
var crashes = new CrashReporter(transport, new CrashReporterOptions {
Dir = Path.Combine(config.DataDir, "crashes"),
ClientVersion = "1.4.2", // the game version, not the SDK's
ReleaseId = patchReleaseId,
});
// Consent. Defaults to false; a settings screen sets these.
telemetry.Enabled = settings.SendUsageData;
crashes.Enabled = settings.SendCrashReports;
telemetry.TrackSessionStart(new SessionStartInfo(ClientVersion: "1.4.2", Os: "Windows 11", Gpu: gpuName,
VramMb: vram, RamMb: ram, Resolution: "1920x1080", Locale: "ja-JP"));
telemetry.Track("level_complete", new Dictionary<string, object?> { ["mode"] = "ranked", ["seconds"] = 312 });
crashes.BeginSession(); // writes the "did the last run end cleanly?" marker
OtomoCrashHooks.Install(crashes); // unhandled exceptions, unobserved task exceptions
await crashes.FlushPendingAsync(); // sends last run's report, if there was one
// On a clean quit:
telemetry.TrackSessionEnd(new SessionEndInfo(lengthSeconds, new Percentiles(6.9, 11.2, 24.0), hitches,
bootMs, patchMs, matchMs, rttP50, rttP95, lossPercent));
await telemetry.FlushAsync(cts.Token); // with a timeout: never hang the quit
crashes.MarkCleanShutdown();
What the SDK does for you: batches events and sends them in the background (never on the main thread);
keeps what could not be sent in a size-capped file on the player's disk and sends it next launch; honours
Retry-After and backs off; sends at most 5 crash reports a day; applies the same scrubbing as the
server; and treats a run that ended without MarkCleanShutdown() as a crash, reporting it on the next
launch. PercentileTracker is a small helper for the frame-time and latency percentiles.
Godot's own error log has no managed event on every 4.x version. Feed it from your logger by calling
OtomoCrashHooks.PushError(message).
Choose which gameplay events to count¶
otomo does not know your game's events and does not guess. You list the ones you want in a small file, and anything not on the list is dropped (and counted, so you can see it happening).
- Copy
deploy/telemetry/events.example.jsontodeploy/telemetry/events.jsonand edit it. - In
deploy/.env, setTELEMETRY_EVENT_SCHEMA_FILE=/etc/otomo/telemetry/events.json. - Run
deploy/scripts/up.sh(the service reads the file when it starts).
{"events": {"level_complete": {"props": {
"mode": {"type": "enum", "values": ["casual", "ranked"]},
"seconds": {"type": "int", "min": 0, "max": 86400},
"won": {"type": "bool"},
"map": {"type": "string", "max": 32}
}}}}
Event names are lower-case letters, digits, _ and .. Each event may have up to 8 properties, each a
bool, an enum (up to 32 listed values), an int or float (a min and max are required), or a short
string. A property the game sends that is not listed, or whose value breaks its rules, is dropped and the
event is kept. otomo keeps a count per event per day, a count per value for every enum and bool
property, and the count, sum, minimum and maximum of every number. It does not keep the strings: they have
no bounded set of values, so they cannot be summarised, and they are where personal data hides. The file is
refused at start-up if it would let a number or a string be unbounded.
Step 3: Read it¶
The admin website pages for this are not built yet (the screens are listed in the service's README). The
staff API is available now, through the same staff gateway as the rest of the admin plane, with a staff
token. Reads need the viewer role; resolving, ignoring and reopening need live_ops.
| Question | Request |
|---|---|
| What is crashing right now? | GET /api/admin/telemetry/crashes?status=open&sort=count |
| Did a fix work? | GET /api/admin/telemetry/crashes?status=regressed |
| Everything about one crash | GET /api/admin/telemetry/crashes/{id}: recent reports, the breakdown by OS, GPU and version, a count per day |
| The player's log | GET /api/admin/telemetry/crashes/{id}/events/{event_id}/log (a plain-text download) |
| How is the game running? | GET /api/admin/telemetry/overview?days=30 |
| Which events happen? | GET /api/admin/telemetry/events?days=30 |
The overview returns sums and counts, so you can add rows together: the mean 95th-percentile frame time
for a version is frame_p95_sum / frame_n, the crash rate is crashes / sessions, hitches per run are
hitch_sum / sessions_ended. A percentile cannot be averaged exactly across runs; what you get is the mean of each
run's own percentile, which is the right number for comparing versions and the wrong one for a promise to
a player.
Everything in these responses that came from a player's machine is untrusted text. It is scrubbed, but
anyone can send anything. Responses mark each object that carries it with "text_encoding": "plain"; whatever
displays it (the admin website, a script, a spreadsheet) must show it as text and never as HTML or Markdown.
The log download is a plain-text attachment for the same reason.
How crashes are grouped¶
Two reports are the same crash when they have the same error type and the same top five frames of your code (frames inside the runtime or the engine are skipped), after the parts that differ from machine to machine are removed: file paths and line numbers, offsets and addresses, argument lists, generic type arguments and the numbers the compiler adds to closures. Each group tracks first and last seen, how many times, the first and last version, and a status:
- open: needs attention.
- resolved: you fixed it. You can say which version fixes it (
{"fixed_in": "1.4.3"}). Once three different players have reported that version or a newer one, it reopens and is marked a regression (one account cannot reopen or silence a group by claiming a version). Without a version, it is resolved as of the newest version that had it, and a report from a strictly newer version reopens it. Reports from older versions still count but leave it resolved. A version string that is not numbers (a nightly name) never reopens anything. - ignored: you have decided not to fix it. It keeps counting and nothing ever reopens it.
Every group keeps its 50 newest reports, so a crash that happens a million times does not grow without bound; the counts and the breakdowns stay exact. Each resolve, ignore and reopen is recorded with who did it.
What is kept, and for how long¶
A sweep runs at start-up and every hour. These are the defaults; the variables are in deploy/.env
(.env.example lists them).
| Data | Kept | Variable |
|---|---|---|
| One row per run of the game | 90 days (dropped a whole month at a time) | TELEMETRY_SESSION_RETENTION_DAYS |
| Daily summaries by version, OS and GPU maker; event counts | 2 years | TELEMETRY_ROLLUP_RETENTION_DAYS |
| Hardware profiles | 1 year after last seen | TELEMETRY_DEVICE_RETENTION_DAYS |
| Individual crash reports | 60 days | TELEMETRY_CRASH_EVENT_RETENTION_DAYS |
| A crash group (counts, breakdowns) | 1 year after it was last seen | TELEMETRY_CRASH_GROUP_RETENTION_DAYS |
| Crash logs | with their report, and never more than 512 MB in total (the oldest go first) | TELEMETRY_LOG_MAX_MB |
The summaries and counts contain no player at all. The per-run rows and crash reports contain the hashed
player id, which is what lets you delete one player's data if you must (the player_hmac column is
indexed). That removal is a database operation today; there is no admin screen for it.
Per-player limits stop one account from flooding the service: 20 batches a minute and 5,000 events a day, and 3 crash
reports a minute and 10 a day. The gateway adds its own per-address limit. All are tunable in .env.
What it costs¶
| Memory | One container, hard-limited to 64 MB (deploy.resources.limits.memory in compose.yaml). Idle it sits around 20 to 30 MB. The only expensive request is a crash upload, and at most four are processed at once, so a crowd of players crashing together gets a polite "try again shortly" rather than an out-of-memory kill. |
| Disk: crash logs | Capped at 512 MB by default (a named volume, telemetry-data). |
| Disk: database | About 250 bytes per run plus indexes: with the default 90 days, a game with 10,000 runs a day keeps about 900,000 rows, on the order of 400 MB with indexes. The summaries and crash groups are small by comparison. Lower the session retention to shrink it. |
| Network | A batch is a few hundred bytes to a few kilobytes every half minute of play; a crash report is at most 400 KB. |
| Other services | Postgres and Valkey (database 4, counters only). It is included in the nightly backup, because the backup dumps every database. |
When something is wrong¶
| You see | Likely cause |
|---|---|
up.sh stops: "the telemetry profile is on and ... is missing" |
You turned the profile on without running generate-secrets.sh, or without the chown it printed. |
The service never becomes ready; /readyz says "no player key set yet" |
Auth has not exported its key file yet, as for every other service; it clears itself. |
| The game sends events and none appear | The event is not in events.json (see telemetry_events_dropped_total{reason="unknown_event"}), or the player has not agreed, or the run was not sampled. |
429 with Retry-After |
The player hit a per-player limit. The SDK waits and retries; nothing is lost unless the queue's size cap is reached first. |
404 not_available from /api/player/telemetry/... |
The profile is off or the container is not running; the SDK stops retrying. (A 502 means it is running and failing.) |
telemetry_crash_logs_dropped_total{reason="full"} rises |
The log volume is at its cap; raise TELEMETRY_LOG_MAX_MB or lower the crash-event retention. Reports are still stored without their logs. |
The service exposes Prometheus metrics on its metrics port (telemetry_*), including how many events were
dropped and why, rate-limit refusals, crash groups open and regressed, the log volume's size, and when the
retention sweep last ran. It is not on the Prometheus scrape list by default, because an optional service
that is switched off would otherwise page someone; Step 1, item 5 switches its job on for a host that runs it.