An eval case carries the obligations of the log it came from

The erasure request — a customer invoking their legal right to have the company delete everything it holds about them — arrived on a Thursday and was routed to the deletion job that the platform team had built two years earlier and tested twice a year since. It cleared the user's rows from Postgres, purged their conversations from the trace store, dropped their documents from object storage, and evicted the derived embeddings from the vector index. Eleven days later an engineer grepping the monorepo for a template bug found the same person's name, email address and two order numbers sitting in tests/evals/cases/case_2026_02_04_0219.json, which had been committed in February, reviewed in a pull request, and copied onto every laptop that had ever cloned the repository. The company's compliance attestation said the data was gone. It was in 41 working copies, in the CI runner's cache, in the build artifact store, in the read-only mirror the security team used for scanning, and in every commit of the repository's history from February onward.

Nothing went wrong in that story except a categorisation. The evaluation cases were treated as source code because they are text files that live next to code and change through pull requests. They are copies of customer content, drawn from a log that has a retention policy and a deletion path, and copying data does not leave the obligations behind in the original.

Why a repository is the wrong container specifically#

The properties that make git excellent for code are the properties that make it unusable for personal data. History is immutable by design, so removing a value means rewriting every commit since it was added, force-pushing, and then getting every clone re-cloned — a coordination exercise that takes days and has no way to prove completion. Distribution is unbounded by design: the whole point is that everyone has a full copy, including the contractor who finished last quarter and the fork someone made for a spike. Access control is per-repository, so a case containing a health-adjacent support conversation is readable by every engineer with repo access rather than by the four people whose job touches it. And there is no expiry mechanism at all — nothing that deletes an object once a stated time to live, or TTL, has elapsed — so the "delete after 90 days" line in the retention policy has nowhere to attach.

Against that, the reasons teams put cases in the repository are real and should be preserved by whatever replaces it. Cases are versioned with the code that consumes them, so a checkout from March runs March's suite. Changes go through review, which is how a bad expected outcome gets caught. And CI can read them with no credentials and no network, which keeps the test job fast and boring.

The arrangement most teams settle on is the second or third column for the cases themselves plus a manifest in the repository: a small file holding the dataset version, the content hash of each case, and the expected outcomes, with the payloads fetched at test time. Reviewers still see expectation changes in a diff, CI still pins to a version, and the customer content lives somewhere with a lifecycle rule and an access log. Langfuse's user-feedback and dataset model is a reasonable reference for the third column, since it keeps the case joined to the trace it came from, which is exactly the join a deletion request needs.

The manifest is worth describing concretely, because it is the piece that makes the split arrangement feel as ergonomic as the repository did. A file at evals/contracts/manifest.json holds the dataset id, a version, and one entry per case containing the case id, the content hash of the payload, the cluster id, the expected outcome, and the redaction state. Everything a reviewer needs to argue about is in that file and shows up in a diff; everything that is customer content is behind a fetch. The test harness resolves the manifest at run time, pulls payloads from the dataset store, and fails loudly if a hash does not match, which catches the case where somebody edited a payload without going through review. A cached copy on the CI runner keeps the job fast, with the cache keyed by content hash and given a lifetime shorter than the retention policy so that a deleted case cannot survive in it for long.

Deletion has to fan out, which means an index#

A deletion path that covers the eval set requires knowing which cases came from which subject, and that is a piece of infrastructure rather than a policy statement. The provenance block from chapter 4 already carries a tenant_hash and a trace_id; what it needs in addition is a subject reference that the deletion service can query.

The tombstone — a marker left in place of the deleted case, recording that something was there without recording what — matters. A suite that silently shrinks from 640 to 637 cases produces a pass-rate change that nobody can explain six months later, so the deletion writes a record — case id, cluster id, date, reason erasure — that carries no customer content and lets a later analysis reconstruct why the denominator moved. This is also the mechanism that keeps chapter 4's coverage numbers honest, since a cluster can lose its only case to an erasure request and the suite should be able to notice that the behaviour is now untested.

What redaction leaves usable, and what it destroys#

Most cases do not need the real values, which is what pseudonymisation trades on: every name, address and order number is swapped for a stable stand-in rather than struck out, so the case still reads as a piece of English and still runs. A question about a delayed refund exercises the same model behaviour whether the customer is Marcus Adeyemi or a surrogate, provided the surrogate is applied consistently: if the name appears in the user's message, in a retrieved chunk, and in the expected output, all three occurrences have to map to the same replacement or the case stops testing coreference — the model's tracking of which later mention refers to which earlier one — and starts testing something incoherent.

import hashlib, json, re

PERSON_POOL = ["Alex Rivera", "Priya Nandan", "Tomas Weiss", "Nia Okafor"]

def surrogate(kind: str, value: str, salt: bytes) -> str:
    """Stable per-case pseudonym: same input value -> same surrogate."""
    h = hashlib.blake2b(value.lower().encode(), key=salt, digest_size=8).hexdigest()
    if kind == "EMAIL":
        return f"user_{h[:8]}@example.invalid"
    if kind == "ORDER":                       # format-preserving: parsers still work
        return "ORD-" + "".join(str(int(c, 16) % 10) for c in h[:9])
    if kind == "PERSON":
        return f"{PERSON_POOL[int(h[:4], 16) % len(PERSON_POOL)]}"
    raise ValueError(kind)

def redact(case: dict, entities: list[tuple[str, str]], salt: bytes) -> dict:
    text = json.dumps(case)
    for kind, value in entities:
        text = re.sub(re.escape(value), surrogate(kind, value, salt), text, flags=re.I)
    out = json.loads(text)
    out["redaction_state"] = "pseudonymised"
    out["redaction_fidelity"] = "format_preserving"
    return out

Format preservation is the detail that decides whether a redacted case still exercises the code path it was captured for. An order number replaced by [REDACTED] breaks the regex in the tool that looks up orders, so the case now fails for a reason invented by the redaction, and a future engineer spends an afternoon on a bug that does not exist. Replacing it with ORD-483920117, matching the real format, keeps every parser, validator and prompt instruction operating on the same shape of input.

There is a class of case where redaction destroys the thing being tested, and pretending otherwise is how sanitised suites become quietly useless. A failure caused by an apostrophe in a surname breaking a template, by an order number from an acquired company that has eleven digits instead of nine, by a date written in a locale format the model misreads, or by a document whose formatting is the whole problem — in each of these the raw value is the test input. On the contracts product from the last chapter, of 640 cases, 512 pseudonymised cleanly, 96 needed hand-checking because the entity appeared inside retrieved document text where the surrogate broke sentence structure, and 32 depended on the real value and could not be sanitised at all.

Those 32 go into a restricted tier: a separate dataset with named access for four people, a 90-day TTL rather than the standard retention, no copy in CI, and a scheduled job that runs them and reports only aggregate pass/fail into the main dashboard. The cost is that a failing case in that tier is harder to debug, because most engineers cannot look at it. The alternative is either discarding the hardest 5% of the suite or holding raw customer data at low protection for years, and both of those are worse trades.

Cases have a way of budding off further copies, and the fan-out has to reach those too. A failing case gets pasted into an incident channel so the on-call engineer can see it. A particularly instructive one becomes a few-shot example in a prompt template, at which point the customer's question is in the production system prompt on every request and in the template's version history. A batch of them is used to fine-tune a small classifier, which is a copy that cannot be surgically removed from the resulting weights at all. Each of these is a real and useful thing to do, and each needs to be a deliberate step with its own record rather than a copy-paste. The practical rule that keeps this tractable is that anything derived from a case must either be built from a fully synthetic or pseudonymised variant, or be registered in the subject index like the case itself, and few-shot examples in production prompts should be rewritten by hand rather than lifted, which takes ten minutes and removes the entire problem.

Access is the other thing that decays quietly. A dataset created with four named readers has eleven of them a year later, because people were added during incidents and never removed, and nothing in the system prompts a review. A quarterly access review that lists every reader of the restricted tier and requires a named owner to re-approve each one is dull, takes twenty minutes, and is the only mechanism that reverses the drift. The bucket access logs from the table above are what make that review meaningful, since a reader who has not opened a case in six months is an easy removal and a reader who opens them weekly is doing work that should be visible.

The two arguments that sound principled and are not#

The first is that the terms of service permit using customer data to improve the service, so this is settled. Permission to process is not the same thing as exemption from erasure, and the ToS clause governs whether you may build the eval set at all, while the retention schedule and the deletion path govern what happens to it afterwards. A team that has the first and not the second has a legal basis for holding data it cannot delete, which is a worse position than having neither.

The second is the recommendation to avoid all of this by generating the suite synthetically. Synthetic cases have genuine uses — they fill clusters that production has not produced enough examples of, and they let you test behaviours that would be unethical to wait for — and they cannot substitute for the set, because the entire reason for mining production was that real traffic contains input distributions nobody would have thought to write. The formatting quirks, the half-sentences, the pasted email threads with three levels of quoting, the questions that presuppose something false: those are what the parser broke on, and a generated corpus reflects the generator's idea of a user rather than a user. A suite built only from synthetic cases converges on testing the failures you already imagined.

What the governance work buys is a set that can keep growing without becoming a liability, and by this point the pipeline is producing one: candidates filtered, deduplicated, given expected outcomes, stored where they can be deleted. The remaining question is what it means when that set passes. Sixty cases came out of last month's queue, the prompt was iterated until every one of them was green, and the team is ready to ship on the strength of it.