User-data deletion

How deletion works today: an admin deletes a user in the CIAM admin UI. CIAM publishes the user's UUID to an SNS topic (ciam-user-deleted), then deletes the Cognito identity. duunitori5 subscribes and anonymizes the matching account: email and name are overwritten, applications are anonymized in place, saved searches, favorites, and the profile are deleted. On the happy path, for a core duunitori.fi account, this genuinely works.

The audit (01-current-state.md — every claim verified against code and cited to file and line) found the holes below. Labels like C2 or H7 are the audit's finding IDs — C critical, H high, M medium, L low, numbered within each tier — each one is indexed in 01-current-state.md if you want the full evidence. These are the ones we propose to fix now:

• Jobbland job-seekers' saved searches — plaintext email plus the full search profile — survive deletion forever. The delete code queries a database field that doesn't exist, so it throws on every single call. Worse, most jobbland users authenticate through CIAM without ever getting a local user row, so the fallback delete-by-email path never runs for them either. Jobbland's primary audience is exactly these users. (Finding C2 in the audit.)
• The handler reports success even when deletion fails. The exception above is caught by a catch-all and the endpoint returns 200, so nothing retries and nobody notices. Silent permanent PII retention, reported as success. (M7.)
• Deletion is neither atomic nor idempotent. Email and name are anonymized last, after all the related deletes. A crash or pod restart midway leaves the real email and name in place — and every retry then dies at the same spot (a profile delete that can't run twice), so the PII is stranded permanently. (H2.)
• Deleting a saved search copies it into an archive keyed by an unsalted SHA-256 of the person's real email. Hashing a known email and matching it against the archive re-identifies the person — so this retained "anonymous" behavioral data is not actually anonymous, and it has no retention limit. (H7.)
• Application confirmation rows keep the applicant's plaintext email forever. No code path — neither the deletion flow nor the age-based application anonymization — ever touches them. (H8.)
• Two smaller hygiene gaps: CIAM never removes its own {uuid, role} permission row for a deleted user (L1), and the deletion handlers log the user's still-linkable UUID after deletion (L3).

The audit also found serious problems we are not fixing in this project — headline ones: the deletion endpoint itself is forgeable by an unauthenticated caller; the moonshot ATS receives no deletion signal at all; and PII copied to third parties (the BigQuery warehouse, OpenAI, Mailchimp, Intercom, Kombo) never gets a delete signal. Where that line sits and why is covered under the proposal and "Out of scope" below.

Why this matters beyond code quality: these are GDPR Article 17 (right to erasure) failures. A deleted user's plaintext email and profiling data demonstrably persisting — in some cases indefinitely, with the deletion reported as successful — is a legal and compliance exposure, and it is provable from our own database.

Target, in plain terms: when a deletion event arrives, duunitori5 fully erases or anonymizes that person's PII — atomically, idempotently, and loudly on failure — and anything we retain afterwards can no longer be tied back to the person. Plus one small cleanup in CIAM's own database. Full target design and decision log: 02-target-state.md.

Each written so you can disagree with it:

• Keep anonymize-in-place for the user row, over hard-deleting it. The team wants to retain de-identified behavioral data (preferences, recommendations, saved-search shapes) for pattern analysis; a hard delete would cascade all of it away. The honest trade-off: the anonymized row still carries the CIAM user id (a deferred finding, M2), so "cannot be tied back" is not fully met until that and the warehouse copies are addressed. We state that openly rather than pretending anonymization is complete.
• Drop the email-hash column from the saved-search archive entirely, over salting/HMAC-ing it. The column is written but never read anywhere in the codebase (verified — no query touches it), so no analytics value is lost, and dropping it removes the historical hashes in the same migration. Salting would have kept a lookup key nobody uses and required a backfill.
• Fail loudly — atomic transaction, idempotent re-runs, non-2xx on any failure — over today's swallow-and-continue. For erasure, a swallowed failure is the worst outcome: PII retained, success reported. Trade-off: the SNS topic has no dead-letter queue (deferred finding H1), so a persistently failing event is retried ~10 times over ~40 minutes and then dropped. That's strictly better than silent success, and the delivery layer gets hardened when that deferral lifts.
• Scrub confirmation emails inside the existing application-anonymization routine, over a separate sweep. Co-locating it means both triggers — the deletion path and the time-based application-anonymization command — cover it automatically.
• Accept, and only document, the Cognito bypass (H11), over a code fix. A user can technically delete themselves directly against Cognito, and an operator can delete via the AWS console — both skip the SNS fan-out, so downstream PII is never anonymized. We can't selectively revoke the OAuth scope enabling self-service deletion (it's all-or-nothing and also required for terms-and-conditions self-service updates), and the robust fix — a Cognito↔database reconciliation job — overlaps the deferred delivery-reliability work. So the mitigation is a written operating rule: all user deletions go through the CIAM admin endpoint. Reassuringly, the existing ciam-ui delete button already routes through that endpoint, so the normal UI path does not bypass anything.
• Forward-fix only; remediating already-stranded historical data is a separate follow-up project. Everyone deleted before these fixes still has stranded PII (jobbland saved searches, confirmation emails). Cleaning that up needs its own analysis — and likely legal input on how far back to go — so bundling it here would stall the forward fixes.

Where the deferral line is drawn, and why:

• The forgeable deletion endpoint (security bundle — findings C1/H3/M3/L2) is deferred by Petri as project owner. This is a prioritization call — ship the erasure fixes first, with the security bundle explicitly not startable right away — not a technical blocker; no lift trigger or date has been set, and no deeper rationale is on record. It is flagged as the most serious finding of the whole audit — anyone who learns a user's CIAM id can forge a "delete this user" event against an endpoint that neither authenticates callers nor properly validates the SNS signature — and it should be first in line when the deferral lifts. Question 5 below asks you to pressure-test exactly this call.
• Everything moonshot is deferred because moonshot has no CIAM accounts today; the CIAM integration is in progress, and the erasure listener is already planned as a phase of that project. Building erasure propagation before the identity link exists would be building on sand.
• Third-party copies (conversation data sent to OpenAI by Josi, the AI chat assistant on duunitori.fi; Mailchimp newsletter subscriptions; Intercom support contacts; applicant data forwarded to external ATSes via Kombo) each need an outbound delete call that doesn't exist; they're deferred until those integrations are next touched. Intercom is flagged as the easy first pick — its API supports permanent contact deletion.
• The BigQuery warehouse (raw email + IP retained, no deletion propagation) is owned by the external data-warehouse partner; it's flagged to them, not a workstream of ours.

Four independent workstreams (03-roadmap.md has the full detail). Each code workstream is its own branch off the repo's base branch, reviewed, and taken to an open PR — nothing merges without human review. Effort: S ≈ hours–1 day, M ≈ several days.

Order and dependencies: the three code workstreams touch disjoint files (two of them in duunitori5, one in ciam), so they can run in parallel with no merge coupling. If serialized, WS1 goes first — it carries the critical bug. WS2 holds the only schema change (the column drop); no ordering constraint against the others.

Rollout and rollback: deletion is a low-frequency, admin-triggered operation, so the atomic transaction poses minimal lock risk. WS1's fail-loudly change means errors that used to silently return 200 will now surface as SNS retries — expect noisier deletion-handler logs right after rollout; that's the intended behavior, and we monitor those logs. WS1 and WS3 carry no schema changes, so reverting either is a plain code rollback. WS2's column drop is the one hard-to-reverse step — it's gated on re-verifying, immediately before merge, that nothing in or out of the repo reads the column. New regression tests (CIAM-only user's saved searches gone; retry after partial failure leaves no real email/name; re-running an event is a safe no-op) are the main safety net, since existing coverage on these paths is thin.

Done means: a deletion event anonymizes/deletes a user's core and jobbland PII atomically, idempotently, and for CIAM-only users, failing loudly on error; the archive keeps no re-identifier; confirmation emails are scrubbed; CIAM cleans its own row; the operating rule is documented; PRs open (not merged).

Technical:

• No dead-letter queue behind fail-loudly. After ~10 SNS retries a persistently failing deletion event is dropped — visible in logs, but nothing alarms on it. We judged this acceptable until the deferred delivery-reliability work; it is a judgment, not a proof.
• The column drop is irreversible. We verified nothing in the repo reads the email-hash column, but an out-of-repo consumer (a warehouse query, ad-hoc analytics) can't be ruled out from the codebase alone.
• Residual re-identification vectors remain by design. Anonymized rows keep the CIAM user id, and the warehouse keeps raw emails — so the "retain behavioral data only if unlinkable" principle is only partly satisfied until those deferred/external items land.

Business / compliance:

• The most serious finding is in the deferred pile. The deletion endpoint being forgeable by any unauthenticated caller is both a data-destruction risk (anyone can erase a victim's account) and arguably a bigger GDPR exposure than anything we're fixing.
• Historical stranded data stays stranded. These fixes are forward-only; every user deleted before them still has plaintext email in jobbland saved searches and application confirmations.
• Deleted users' PII keeps sitting at third parties in the meantime. Until each third-party deferral lifts, Mailchimp, Intercom, OpenAI, and Kombo retain a deleted user's data indefinitely — and the trigger ("when each integration is next touched") has no date or owner attached, so this exposure has no scheduled end.
• An accepted risk is only as good as its enforcement. The Cognito-bypass rule is a written convention, not a technical control.

The concrete questions we want the team to answer — this is the discussion agenda.

Technical:

1. Does anything outside the duunitori5 repo read the saved-search archive (ArchivedJobwatch), especially its email-hash column — a warehouse query, a dashboard, ad-hoc analytics? We need a confident "no" before the drop migration merges.
2. Is SNS's built-in retry (no dead-letter queue, no alert) acceptable as the interim failure story for deletion events, or should WS1 ship with a minimal alert on deletion-handler 5xx responses?
3. Are there any other ways user deletions get triggered today — support scripts, admin habits, console access — that the WS4 operating rule needs to name explicitly?

Business:

1. Is the accepted-risk list actually acceptable — in particular, handling the Cognito self-service/console bypass with a written rule instead of a technical control? Who signs off on that acceptance?
2. Is the deferred security bundle — an unauthenticated, forgeable deletion endpoint — genuinely deferrable? What event or date lifts the deferral, and who owns scheduling it?
3. Who owns kicking off the historical-remediation follow-up (scrubbing already-stranded emails from past deletions), and does legal need to weigh in on how far back we go?
4. Should we engage the data-warehouse partner about BigQuery erasure (raw email + IP retained with no deletion propagation) now, or wait until our own side is fixed?
5. Is "when the integration is next touched" an acceptable trigger for the third-party delete signals (Mailchimp, Intercom, OpenAI, Kombo), and who owns tracking that none of them is forgotten? Intercom is flagged as the easy first pick — its API supports permanent contact deletion.

Deliberately not in this project, tracked in the reference docs (README.md holds the full disposition lists):

• SNS ingest security bundle — the forgeable/unauthenticated deletion endpoint, missing topic validation, SSRF and replay issues. Deferred on the project owner's prioritization call (erasure fixes ship first; no lift trigger or date set — see Question 5); top priority when the deferral lifts.
• All moonshot erasure — no deletion propagation into the ATS, four email tables never scrubbed by any path, recruiter accounts with no deletion path, age-out gaps. Deferred pending the CIAM↔moonshot integration, which already carries the erasure listener in its own roadmap.
• Third-party delete signals — OpenAI (Josi conversations), Mailchimp (newsletter), Intercom (support contacts; first candidate when picked up, as its API supports permanent deletion), Kombo (applicant data forwarded to external ATSes).
• BigQuery warehouse erasure — owned by the external data-warehouse partner; flagged to them.
• Other deferred duunitori5/CIAM items — SNS delivery loss (no dead-letter queue), CV files surviving in S3, the retained CIAM user id on anonymized rows, the edge case where a deleted email address is reused before a delayed deletion runs (so a retry could delete the wrong person's data), deletion events arriving out of order, legacy users who predate CIAM and have no erasure path, and the case where CIAM has already published the deletion event but the Cognito delete itself then fails.
• Accepted — retaining de-identified behavioral data (conditional on it staying unlinkable), and the Cognito-bypass handled by rule (WS4 documents it).
• Historical remediation of already-stranded data — separate follow-up project.
• DSAR / data-export completeness (GDPR Article 15) beyond noting where export and erasure diverge; likewise database backup/snapshot retention windows, which the audit did not examine at all.