observability

Observability — progress

implementing15/38 tasks · updated 2026-07-24
Brief

All five code PRs are merged; the docs PR merged earlier. Activation has started: the NR license key is set in SOPS and the Django backend SENTRY_DSN is set. What remains is the rest of the activation checklist in runbook.md (Sentry projects/DSNs for the 5 newly-covered Next apps + caps, the k8s NR deploy/verify, Slack channels, alert wiring, Pingdom, SNS migration) plus a decision on the closed-unmerged duunitori-tests PR.

Merge queue

5/6
  • duunitori-next #3831 — Sentry factory + app coverage; merged 2026-07-22. Set Sentry per-project caps + spike protection before deploy/DSN activation (jobbsafari.se jumps to 100% error capture)
  • duunitori5 #5527 — Django Sentry SDK; merged 2026-07-23 (dormant until SENTRY_DSN is set)
  • duunitori-kubernetes #283 + duunitori5 #5528 — the NR license-key pair; both merged 2026-07-23. License key now set in SOPS (deploy render unblocked); rotation + deploy pending confirmation
  • duunitori-kubernetes #282 — NR entity consolidation; merged 2026-07-23
  • duunitori-tests #4 — severity-based Slack routing; closed unmerged 2026-07-21 (no reason recorded) — needs a decision: reopen as-is or re-scope the synthetic-alert routing
  • dt-docs #1 — runbook + ops checklist + docs; merged 2026-07-17

Phase 0 — Discovery & quick wins

2/7
  • 0.1 Confirm the moonshot "Dialog" NR feed — needs a human in the New Relic UI (entity type / data source), then the AWS console / infra partner
  • 0.2 Inventory SNS topics/alarms + owners — needs AWS console access; gates the SNS migration (3.6)
  • 0.3 Sentry + NR volume/cost baseline — needs the vendor UIs; gates the hard-cap levels (1.8/2.3) and the sampling revisit (4.2)
  • 0.4 Request read access to the infra partner's alert channel — needs a human to ask the partner
  • 0.5 Create #platform-alerts-critical / #platform-alerts-warning + incoming webhooks — needs a Slack admin; prereq for all alert wiring (3.2–3.5)
  • 0.6 NR hygiene: license key → secret, dead crawlers/analytics configmaps deleted, duplicate web-entity config collapsed (#282 deletes the unused base configmap_newrelic.yaml; the image-baked newrelic.ini stays the single source) — duunitori5#5528, duunitori-kubernetes#283, duunitori-kubernetes#282 (all merged 2026-07-23)
  • 0.7 Fix broken jobbland.no Sentry — folded into the factory, duunitori-next#3831 (merged 2026-07-22)

Phase 1 — Sentry pillar

5/8
  • 1.1–1.4 Shared observability factory in next-shared; 6 configured apps migrated; jobbland.no fixed + 4 missing apps added; Speed Insights standardized — duunitori-next#3831 (merged 2026-07-22)
  • 1.5 Django Sentry SDK: errors + low-sample tracing + structlog integration (structlog flows through stdlib logging; the SDK's LoggingIntegration turns logger.error(...) into Sentry events) + PII scrubbing — duunitori5#5527 (merged 2026-07-23)
  • 1.6–1.7 Sampling (~100% everywhere except duunitori.fi + jobbsafari.no at 1%) and FE→BE trace propagation, code side — in #3831 + #5527 (both merged; end-to-end traces need DSNs set)
  • 1.5 Activation: Django backend SENTRY_DSN set — the backend SDK is now live (verify PII scrubbing on a test event)
  • 1.6/1.8 Activation: create Sentry projects + set DSNs for the 5 newly-covered Next apps (jobbland.no/se, ciam-admin, duunitorigroup, keeparo-redirect), enable spike protection, per-project hard caps + near-cap alerts — human console work; caps must precede enabling error capture (jobbsafari.se jumps to 100%)
  • 1.9 Cron check-ins coded for check_expiry + clean_up_old_data — in duunitori5#5527 (merged 2026-07-23)
  • 1.9 Create the two Sentry cron monitors (schedules/margins matching the k8s crontabs) — Sentry UI work
  • Regenerate duunitori5 requirements.txt via uv pip compile to confirm the hand-added sentry-sdk pin
  • job-ad-generation Sentry — deferred: App-Router app, outside the Pages-Router factory's scope; keeps Langfuse, Sentry via instrumentation.ts in a later chunk

Phase 2 — New Relic rationalization

2/6
  • 2.1 Cron/background work folded onto the existing Background Tasks entity; dead configmaps deleted — duunitori-kubernetes#282 (merged 2026-07-23)
  • 2.1 newrelic.licenseKey set in the SOPS deploy secrets — unblocks the deploy render (was failing by design without it)
  • 2.1 Rotate the previously-exposed key in NR (if not already) + run the k8s deploy — pending confirmation
  • 2.1 Retire/re-point NR dashboards & alerts for the former Tagging/Analytics/Crawlers entities — NR UI, after cutover
  • 2.2 Verify after cutover: web/admin/cron entities still report, log forwarding intact — post-deploy check
  • 2.3 NR ingest hard cap + near-cap budget alert — NR UI; needs the 0.3 baseline

Phase 3 — Alerting rationalization

0/7
  • 3.1 Define baseline alert conditions (availability, backend 5xx/error spike, latency, throughput drop, frontend error spike, missed critical cron, synthetic failure) — pending; needs both pillars emitting
  • 3.2 Wire NR alerts → #platform-alerts-* by severity — NR UI; unblocked by 0.5 + 3.1
  • 3.3 Wire Sentry alerts → #platform-alerts-* by severity — Sentry UI; unblocked by 0.5 + 3.1
  • 3.4 Extend Pingdom simple-uptime to all production domains + route into the channels — Pingdom console; unblocked by 0.5
  • 3.5 Severity-based Slack routing for synthetic failures — built in duunitori-tests#4, but the PR was closed unmerged 2026-07-21; needs a decision (reopen or re-scope)
  • 3.5 Set SLACK_WEBHOOK_URL_CRITICAL / SLACK_WEBHOOK_URL_WARNING GitHub Actions secrets in duunitori-tests — unblocked by 0.5 (moot until the #4 decision)
  • 3.6 Migrate SNS: app alarms → NR, the rest → native SNS→Slack (AWS Chatbot), retire email-forwarding hops — unblocked by the 0.2 inventory

Phase 4 — Harden & finalize

1/4
  • 4.1 Tune thresholds + finalize caps after a couple of weeks of real signal
  • 4.2 Revisit duunitori.fi frontend error sampling given observed volume/cost
  • 4.3 "Which tool for what" runbook + ops checklist written — runbook.md, merged via dt-docs#1 on 2026-07-17
  • 4.4 Retire redundant legacy alert paths once the new ones are proven
Log
  • 2026-07-24 — Activation started: NR license key set in SOPS (deploy render unblocked) and Django backend SENTRY_DSN set (backend Sentry now live). Remaining Sentry activation is the 5 newly-covered Next apps' projects/DSNs + caps; k8s NR deploy + post-deploy verify and key-rotation confirmation still pending.
  • 2026-07-24 — All five code PRs merged (verified on GitHub): duunitori-next #3831 on 07-22; duunitori5 #5527 + #5528, duunitori-kubernetes #282 + #283 on 07-23. Code work is complete; remaining work is the manual/ops activation checklist (runbook.md) plus the duunitori-tests #4 reopen/re-scope decision. NR license key still needs SOPS+rotation before the k8s deploy renders.
  • 2026-07-22 — PR-state audit against GitHub corrected the board: dt-docs #1 was merged 2026-07-17, and duunitori-tests #4 was closed unmerged 2026-07-21 (no reason recorded — needs a reopen/re-scope decision). Five code PRs remain open and CI-green.
  • 2026-07-22 — Review round (three independent reviewers) applied to brief/progress/README. Two board corrections confirmed against the PRs: the duplicate web-entity NR config is collapsed by #282 (it deletes the unused base configmap_newrelic.yaml), and #5527 does deliver roadmap 1.5's structlog integration (stdlib LoggingIntegration captures structlog error logs) — 0.6 and 1.5 now fully done, pending merge.
  • 2026-07-21 — Docs migrated to the brief/progress two-layer format; this board replaces the earlier (undated) progress log. Earlier events, newest first:
    • earlier: post-review follow-ups — NR cron consolidation retargeted onto the existing background-tasks entity instead of a new "Jobs" entity; jobbsafari.no client error sampling set to 1% to match duunitori.fi; target-state/roadmap/runbook updated; CI + CodeRabbit re-verified green on #282 and #3831.
    • earlier: duunitori-kubernetes PR split in two on review request — #282 (entity consolidation, no secrets dependency) and #283 (license key → Secret); #283 + duunitori5 #5528 identified as the deploy-together pair.
    • earlier: wrap-up — CodeRabbit reviews re-checked (no new comments), implementation summary written, all PRs confirmed CI-green.
    • earlier: initial implementation — PRs opened: duunitori-next #3831, duunitori5 #5527 + #5528, duunitori-kubernetes #282, duunitori-tests #4, dt-docs #1; CodeRabbit fixes applied along the way.