Observability

A code/config audit across duunitori5, duunitori-next, moonshot and duunitori-kubernetes (full detail in 01-current-state.md) found three overlapping problems.

Fragmented tooling — no single place to look. Four telemetry destinations, split by stack:

• duunitori5 (Django): backend errors and APM go to New Relic. Its legacy JS frontend (Django templates — one of duunitori.fi's two frontends, alongside the Next.js app below) reports to Sentry, but samples errors at 2%, so ~98% of its client errors are dropped.
• duunitori-next: Sentry, where configured (see below), with client errors sampled at 1%; console logs are stripped from production builds.
• moonshot: Sentry on the frontend; the Go backend has no error tracker in the codebase — production errors are only findable by grepping CloudWatch logs. (A New Relic entity called "Dialog" does receive moonshot backend data, but through no mechanism found in any repo — see known unknowns below.)

Following one user-facing error from browser to backend means jumping tools, and backend Python errors have no triage workflow — they sit in New Relic as APM context that nobody works through.

Coverage holes. Of the 13 apps in duunitori-next, only 6 have working Sentry. jobbland.no has the Sentry dependency installed but no config files — the integration is dead. ciam-admin, duunitorigroup, jobbland.se and keeparo-redirect have no error tracking whatsoever. (The remaining two of the 13: storybook isn't a production app, and job-ad-generation uses Langfuse by design — see Out of scope.) The Django backend has no Sentry. Every configured app carries its own copy-pasted Sentry config, which is exactly how this drift happened.

Alerting is five uncoordinated paths into Slack — and only one of the paths our team owns lives in source control:

There is no severity model, no consistent channel, no cross-source view, and the closest thing to paging is Pingdom texting specific people. On top of that sit hygiene issues: the New Relic license key is hardcoded in two repos (and in git history), the k8s charts carry dead NR config for two entities that no longer exist plus a duplicate config for the live web entity, and Grafana still runs its default admin password.

If we do nothing: production errors on five apps stay invisible (the four with no tracking, plus the dead jobbland.no integration), the flagship site's legacy frontend keeps discarding ~98% of its client errors, incident triage stays tool-hopping, and the alert paths keep living in vendor UIs that can't be reviewed, versioned, or reproduced if an account or person disappears.

The target is a two-pillar model (full decision log in 02-target-state.md):

• Sentry — the error system of record. Every production app — all Next.js apps, duunitori5's JS frontend, and now also the Django backend — reports errors to Sentry, with distributed tracing from browser through Next to Django, session replay (PII-masked), and check-in monitoring for critical cron jobs.
• New Relic — backend telemetry. duunitori5 APM, traces and log forwarding, on a cleaned-up three-entity model (Web, Admin, Background Tasks). NR's error collector stays on, but only as transaction context — triage happens in Sentry.
• Infrastructure observability stays with the external infra partner (Prometheus/Grafana/AlertManager on EKS).
• Alerting: everything our team owns routes into two Slack channels by severity — #platform-alerts-critical (page-worthy) and #platform-alerts-warning (FYI).
• "Single pane of glass" is deliberately realized as crisp ownership plus a documented "which tool for what" table (in runbook.md), not one literal screen.

Each with its why and what was rejected:

• Two pillars rather than one vendor. Consolidating everything into New Relic would lose Sentry's Next.js/React tooling and session replay; consolidating into Sentry would forfeit deep backend APM we already have and pay for. And since infra metrics sit in the partner's Prometheus regardless, a single vendor never actually delivers one pane — it's a large migration for little gain.
• Sentry becomes the system of record for all errors, backend included. The rejected alternative is the status quo — backend errors in NR, frontend in Sentry — which is precisely the split that forces tool-jumping. One triage queue across frontend and backend. Backend events are PII-scrubbed server-side (no cookies, auth headers, or user emails), verified on a test event before rollout.
• A shared observability factory in next-shared, enforced by code review — no CI gate. Per-app copy-paste config is the root cause of the drift (one broken app, four missing); one shared factory fixes it at the root. The deferred alerts-as-code work would partly backstop drift later: Terraform-declared per-app Sentry alert rules would make an app missing its alert coverage visible as a gap in reviewable config.
• ~100% error capture everywhere except the high-traffic duunitori.fi and jobbsafari.no frontends, which stay sampled. Today's 1–2% sampling on low-traffic apps is a blind spot, not a saving. Concretely: duunitori.fi's Next.js app stays at 1% client-error sampling (unchanged) and jobbsafari.no was set to 1% to match; duunitori.fi's legacy duunitori5 JS bundle keeps its current 2% for now; every other app moves to ~100%. The high-traffic exceptions keep sampling to avoid a quota blow-out — revisit once real volume is known (Phase 4).
• Sentry surface = errors + tracing + session replay. No Sentry Logs, no profiling. Sentry Logs is explicitly turned off: logs stay in CloudWatch (store) and New Relic (context next to APM). Adopting Sentry Logs would create a third log destination and blur the pillar boundary this project exists to draw.
• New Relic slims to three entities — Web, Admin, and the existing Background Tasks entity — folding the tagging/analytics/crawlers cron workloads in, and deleting two configmaps for entities that no longer exist in NR. Rejected: keeping the six-entity split (two are already dead; per-workload entities fragment dashboards and alerts) and a single entity for everything (batch jobs would contaminate web APM). Reusing the existing background-tasks entity preserves its history, dashboards and alerts.
• Two severity-tier Slack channels; no PagerDuty/Opsgenie. A paging product was rejected this phase because there is no on-call rotation to feed it — Pingdom SMS remains the only paging. Per-domain channel routing was rejected until team structures actually diverge.
• Hard budget caps, with headroom, spike protection and near-cap alerts. Predictable spend was prioritized over never-drop-data; the mitigation is caps set above expected volume plus alerts before the ceiling, so a real incident doesn't silently hit the cap.
• Uptime: Pingdom extended to all production domains; deep synthetic journeys stay duunitori.fi-only. Sentry's uptime product was rejected as a third overlapping uptime tool; multi-site Playwright journeys are deferred as cost > value right now.
• No formal SLOs (pragmatic thresholds), and alerts configured in the vendor UIs for now. SLOs were rejected for this phase because there's no on-call rotation or error-budget process to enforce them — the ceremony would add overhead without changing anyone's behavior. Terraform-managed alerts-as-code is the stated eventual goal, but the bootstrapping cost isn't worth paying this phase — deferred, not dropped.

Five phases (full tables in 03-roadmap.md; effort scale: S ≈ hours–1 day · M ≈ several days · L ≈ 1–2+ weeks):

One sequencing note: the roadmap as planned gated the 100%-capture flip on the Phase-0 cost baseline. Since the code got built first, the proposal is now the other way around: conservative caps and spike protection are set before the factory PR merges, then adjusted once the baseline exists — question 2 below asks whether that's acceptable.

Where it stands now: everything codeable is built and reviewed (all CI-green, CodeRabbit-reviewed). Five code PRs are open across duunitori-next, duunitori5 and duunitori-kubernetes; the docs PR (on petrihanninen/dt-docs — the repo behind this site) is already merged; and the duunitori-tests severity-routing PR was closed unmerged on 2026-07-21 and needs a decision (reopen as-is, or re-scope). The list, dependencies and merge order are in 04-implementation-summary.md. What remains is the merges plus the manual/ops activation checklist in runbook.md: Sentry projects and DSNs, caps and spike protection, the NR key rotation, Slack channels, alert rules, Pingdom checks.

Rollout / rollback story:

• The code is dormant until configured: the Django Sentry SDK no-ops without SENTRY_DSN, the Next factory captures nothing without a DSN, and the synthetic-test routing falls back to today's webhook until the new secrets exist. Nothing activates by merge alone; rollback is reverting a PR.
• The one sharp edge is the NR license-key pair: a duunitori-kubernetes PR and a duunitori5 PR together remove the hardcoded key, so the key must first be set in the encrypted deploy secrets (and the exposed one rotated), then the k8s PR deployed, then the duunitori5 one. The chart deliberately fails the deploy render if the key is missing — so NR can't silently go dark — but CI lint still shows green, so the PRs look safer than they are.

Definition of done: every prod app + the Django backend report errors to Sentry (100% capture except the sampled duunitori.fi and jobbsafari.no frontends); browser→Next→Django traces work; NR runs the three-entity model with no hardcoded secrets; all team-owned alerts land in #platform-alerts-* by severity; Pingdom covers all production domains; SNS is inventoried and migrated; cost is capped.

Technical

• NR license-key deploy coupling. Deploying the duunitori5 key-removal PR without the k8s secret PR — or without the key set in the deploy secrets — takes New Relic dark for the web app. The Helm required guard fails the deploy render as a safety net, but CI passes regardless, so "green PR" does not mean "safe to merge in isolation" here. The old key sits in git history and must be rotated no matter what.
• jobbsafari.se error-volume jump. When the factory PR merges, jobbsafari.se goes from ~0.5% to 100% error capture. Per-project caps and spike protection must exist before that merge — and cap levels are being set before the Phase-0 cost baseline exists, so they will need adjusting.
• Deliberate behavior changes to be aware of: Sentry Logs is turned off for the Next apps (logs stay in CloudWatch/NR); Vercel Analytics was removed from the jobbsafari apps (traffic is already covered by GTM/GA); jobbsafari.no client error sampling was set to 1% to match duunitori.fi.
• End-to-end tracing needs both sides live. Browser→Next→Django traces only complete once both the frontend factory PR and the Django SDK PR are merged and their DSNs are set. Each merges independently; the trace just isn't end-to-end until both are live.
• Cron check-ins have a sharp constraint. The check-in decorator is only safe on management commands run by exactly one CronJob; a command invoked from several schedules (one runs from nine crons today) would collapse all cadences into one meaningless monitor. Expanding coverage needs per-invocation slugs.
• Known unknowns: how the moonshot "Dialog" New Relic entity is fed (every in-repo and on-cluster mechanism has been ruled out; believed to be an AWS/NR-account-level integration set up by the infra partner); what the AWS SNS topics/alarms actually contain; real Sentry/NR volume and cost.

Business / organizational

• Hard caps mean data drops at the ceiling during an extreme incident — accepted for cost predictability, mitigated by headroom, spike protection and near-cap alerts.
• The infra partner owns cluster monitoring and its alert channel, which we currently can't see. The ownership boundary is asserted in this plan, not yet agreed with them.
• Paging remains Pingdom-SMS-to-named-individuals; there is still no on-call rotation. That's a conscious deferral, not an oversight.

The concrete questions we want the team to answer — this is the discussion agenda.

Technical — sanity-check before the merges:

1. The NR license-key procedure: set the new key in the encrypted deploy secrets and rotate the old one → deploy the k8s PR → deploy the duunitori5 PR. Does anyone see a hole? Who performs the rotation, and in what window?
2. Sentry caps: what monthly event/replay budget should the per-project caps encode, given jobbsafari.se jumps to 100% capture at merge? Are we comfortable merging before the cost baseline exists, with caps set conservatively?
3. Sentry Logs is off by design (logs stay in CloudWatch/NR) — does anyone rely on or want Sentry Logs?
4. The sampling exceptions: duunitori.fi's Next.js app and jobbsafari.no stay at 1% client error sampling, and duunitori.fi's legacy duunitori5 JS bundle keeps its current 2% — are those the right exceptions, and are the rates right?
5. Vercel Analytics is removed from the jobbsafari apps — is there any traffic reporting it provided that GTM/GA doesn't cover?
6. Beyond check_expiry (expires job ads past their end date) and clean_up_old_data (retention cleanup of old user data) — two duunitori5 jobs where a silent failure has direct user/data impact — which scheduled jobs are business-critical enough to deserve cron check-in monitors?

Business / process:

1. Who creates and owns #platform-alerts-critical / #platform-alerts-warning, and who needs to be a member?
2. Who approaches the infra partner about read access to their alert channel, the SNS inventory, and the "Dialog" feed question?
3. Is Pingdom SMS to named individuals acceptable as the only paging until an on-call discussion happens — and who should be on that list?
4. Who owns the Sentry/New Relic budget and signs off the cap levels?

• Infrastructure observability. Prometheus/Grafana/AlertManager on EKS is the external infra partner's domain. Our only actions are requesting read access to their alert channel and clarifying the ownership boundary.
• moonshot. No app changes this phase. Its Go backend still has no error tracker; the only action is identifying and documenting the mystery New Relic "Dialog" feed. Onboarding moonshot into the two-pillar model (Sentry Go SDK, NR-vs-partner APM decision) is explicitly future work.
• Deferred deliberately: alerts/config as code (Terraform for both pillars — the eventual goal; vendor-UI config for now); unified cross-stack structured logging and shipping Next server logs off Vercel; multi-site synthetic journeys for jobbland/jobbsafari; formal SLOs and error budgets; a dedicated on-call/paging tool; an aggregated status dashboard; Sentry for job-ad-generation (an App-Router app outside this round's factory scope — it keeps Langfuse for LLM tracing, Sentry added later).