A canary found in a sink is evidence; a unit test is a claim

Four weeks before the tracing incident, the same team filled in a customer security questionnaire. The question read "describe the controls that prevent personal data from being transmitted to sub-processors, and the evidence you hold for their effectiveness." A sub-processor, in the contractual vocabulary the question is written in, is any third party you hand your customer's data to in order to do your job — the model provider, the observability vendor, the error tracker. The answer said that all payloads pass through a redaction service before transmission, that the service covers eleven entity types, and that it is verified by 340 unit tests which run on every commit and are currently green. The customer accepted it. So did an internal review. The answer was true in every particular and the personal data went out anyway, through a span exporter that no test had ever considered, because the question the tests answer and the question the questionnaire asks are different questions.

A unit test over the redactor establishes something conditional: given that this function receives this string, it returns that string. The claim in the questionnaire is universal over paths: for every way a payload can leave this process, redaction happened first. No quantity of tests on the first form adds up to the second, since the second is falsified by a path that never calls the function, and a path that never calls the function is invisible to tests of the function.

Evidence for a universal claim over paths has to be produced by exercising the paths. The instrument is a canary: a payload carrying values you can search for, driven through the real pipeline, followed by a sweep of every place a payload can end up.

Designing a canary that means something when it is found#

The canary carries two different kinds of marker and confusing them is the usual mistake.

A locator token is a high-entropy string placed in an innocuous field — the ticket subject, a metadata key — that identifies which records in a sink belong to this canary run. It is not personal data and it is not expected to be redacted. Its job is to let the sweeper find the run's footprint in a log store holding 40 million lines.

The payload values are synthetic personal data, formatted exactly the way real values are formatted. These are the things that must never appear in a sink, and their design decides what the test can prove. They have to be values the detector recognises easily and reliably, because a canary is an instrument for topology rather than for coverage. If you seed the canary with the Dutch phone format from chapter 1 — a format the detector demonstrably misses — then every run will report a hit at the provider, and the hit tells you what you already know from the recall measurement. Seeding it with an unambiguous, high-confidence value means that any appearance anywhere is unambiguous evidence that the payload reached that sink without passing through redaction at all.

Getting realistic values without creating the exposure you are testing for is a solved problem, because several standards bodies have reserved ranges for exactly this.

CANARY_PII = {
    "email":        "hendrika.vermeulen@canary.invalid",   # RFC 6761 reserved TLD
    "phone_uk":     "07700 900412",                        # Ofcom drama range
    "phone_us":     "+1 202 555 0147",                     # 555-01xx reserved
    "payment_card": "4111 1111 1111 1111",                 # test PAN, valid Luhn
    "iban":         "GB33BUKB20201555555555",              # documentation IBAN
    "person_name":  "Hendrika Vermeulen",
    "address":      "12 Vollenhovestraat, 1015 XX Amsterdam",
}

def canary_payload(run_id: str) -> dict:
    return {
        "ticket_id": f"CANARY-{run_id}",          # locator, not secret, not PII
        "customer_ref": TEST_ACCOUNT_ID,          # a real account, owned by us
        "transcript": CANARY_TEMPLATE.format(**CANARY_PII),
    }

Nothing in that dictionary belongs to a person. The card number is the Visa test PAN — the primary account number from chapter 2 — which passes a Luhn check, the digit checksum every real card number satisfies and the thing a payment form uses to reject a typo, while being declined by every processor. The phone numbers come from ranges regulators set aside so that television dramas do not call real households. canary.invalid cannot resolve. And the canary runs against a real internal test account, so it traverses the same authorisation, the same tenant routing, and the same queue partitions as a customer request rather than a special path.

Sweeping the sinks, including the ones only failures reach#

The sweep is the other half, and it needs an adapter per sink because each one is searched differently.

The two assertions in the last step are a pair and dropping either one makes the test worthless. Searching for the synthetic values and finding none is the result you want, and it is also exactly what you get when the sweeper is querying the wrong index, the canary request failed before it did anything, or the sink's retention window already expired. The locator assertion is the positive control: the run must be found in each sink before its absence of personal data means anything. A canary suite without positive controls degrades into a suite that passes because it is looking at nothing, and it does so silently.

def sweep(run_id: str) -> list[Finding]:
    findings = []
    for sink in SINKS:                       # otel, loki, sentry, kafka_dlq, redis
        records = sink.search(run_id, window=timedelta(minutes=15))
        if not records:
            findings.append(Finding(sink.name, "POSITIVE_CONTROL_MISSING", None))
            continue
        blob = "\n".join(r.raw for r in records)
        for label, value in CANARY_PII.items():
            if value in blob or _normalise(value) in _normalise(blob):
                findings.append(Finding(sink.name, "PII_PRESENT", label))
    return findings

Normalising before comparison matters more than it looks. A phone number written 07700 900412 in the request may be stored as 07700900412 by a system that strips whitespace, or as 07700 900412 after passing through a rich-text editor, and a naive substring search reports clean. The same applies to JSON escaping, base64-encoded message bodies, and gzip: the sweeper has to decode a sink's storage format before searching it, and a sweeper that searches compressed bytes will always pass.

The SINKS list carries the same staleness problem as the boundary enumeration from chapter 1, and there is a partial answer to it. Rather than maintaining the list by hand, derive it from things that change when a sink is added: the egress rules in the service's network policy, the destinations configured in the OpenTelemetry collector, the topics the service is granted produce rights on. A weekly job that diffs the derived set against the swept set will not catch a sink that shares an existing egress path, and it will catch the new analytics exporter somebody wired up on a Friday, which is the realistic failure.

Failure paths are where the canary earns its keep, because they are the paths with no test coverage and the most copying. Forcing them is a matter of making the mock provider misbehave on demand.

Running that matrix in CI takes a compose stack with a local collector speaking OTLP (the OpenTelemetry wire protocol your spans are exported over), a MinIO bucket, a single-broker Kafka, a Redis, and a mock provider that reads its behaviour from a header. On a mid-range CI runner the full six-scenario suite takes about 95 seconds, most of which is the 30-second timeout case, and it is worth running on every change to the ingestion or transport layers rather than only nightly.

The production version is the same instrument aimed at real infrastructure. A scheduled job fires one canary every ten minutes against the production endpoint, waits three minutes for export batching and queue lag, and sweeps by calling the observability vendor's search API, the log platform's query API, a consumer on the dead-letter topic, and a scan of the cache keyspace. Cost is trivial — roughly 4,300 requests a month, each a few hundred tokens against the real provider — and the metrics need a canary tag so the synthetic traffic does not contaminate latency percentiles or usage dashboards. What it catches is the class of defect that no pre-production test can: a new sink added by a platform team, a sampling configuration change that starts recording bodies, a sidecar upgrade that begins mirroring requests.

The assertion that looks like proof and is not#

The tempting shortcut is to put the detector itself at the outbound HTTP client, run it over the request body, and throw if it finds anything. It reads like the strongest possible control, applied at the last possible moment.

As evidence about coverage it is circular. Whatever the detector failed to recognise in the input, it will fail to recognise at the boundary, because it is the same detector applying the same rules to the same string. The assertion therefore passes precisely in the cases that matter — the Dutch phone number sails past on the way in and sails past on the way out — and its green status is not information about the thing it appears to be checking.

As a backstop against topology bugs it is genuinely useful, and it is worth keeping for that reason alone. When an entire unredacted payload reaches the client because a new code path skipped the boundary, the detector will light up with a dozen high-confidence entities at once, and the throw is a real save. The signal to watch is not "did it find anything" but the ratio: if the ingress detector finds entities in 61% of documents and the egress assertion finds them in 0.02%, the pipeline is behaving; if the egress rate climbs towards the ingress rate, something upstream stopped running. That comparison is a monitor, and it is the honest version of the control.

A canary has limits of its own that belong in whatever document the evidence goes into. It proves things about the paths it traverses on the runs it makes, so a defect that only manifests under a partition between two specific availability zones, or on a code path taken by one legacy tenant, will not be found by a canary that never takes it. Coverage of the failure matrix is a design choice you can widen and never complete. What the canary does establish, which nothing else in this topic does, is that on this run, through this pipeline, the payload reached these sinks and did not carry personal data into them.

Every sink in that sweep has one thing in common: you can query it. The provider cannot be swept, cannot be grepped, and does not answer questions about what it still holds — which makes the decision to send it anything a different kind of decision from everything discussed so far.