Safety constraints and residual rates

Everything in this topic so far has treated compliance as a quality metric, where 97.9% is a good number and the remainder is a scatter of small disappointments. Safety constraints break that reading in two places. The residual is not a scatter — at volume it is a steady stream of exactly the event the constraint exists to prevent — and the population the eval sampled from is not the population that produces violations, because safety rules are broken by the unusual tail and, in an adversarial setting, by inputs chosen specifically to break them.

The worked example is a summarisation service that condenses internal support notes so a customer-facing agent can paste the summary into a reply. Some notes are flagged internal_only — fraud review comments, chargeback strategy, the reason an account was silently rate-limited — and the constraint is that content from those notes must never appear in the summary. The eval behind the figures is two thousand frozen cases, five runs per arm, per-rule scoring on all nine rules in the prompt with the other eight flat inside their spread, and a predicate that is mechanical rather than a judge: every 8-gram of every internal-only note, meaning every run of eight consecutive words in it, checked against the summary, with a sampled manual review confirming the automated count.

Two thousand cases put the new rate between 0.20% and 0.79%#

Sixty-two violations in the before arm and eight in the after arm are the raw counts behind 3.1% and 0.4%, and eight is a small number to build a decimal place on. A Wilson interval at 95% — a confidence interval built to stay sensible when the number of observed events is small, where the textbook formula would hand back a range including impossible negative rates — puts the before rate at 2.43% to 3.95% and the after rate at 0.20% to 0.79%. The improvement is solidly real, since the intervals do not come close to touching, while the size of it is known only to within a factor of six. "Eightfold" is one reading of the same data that also supports threefold and twentyfold.

Why an interval of that particular kind, rather than the one from a statistics course, is worth a paragraph, because the shape of the problem recurs wherever a rare event is being counted. The textbook interval takes the observed rate and adds and subtracts a multiple of the square root of the rate times one minus the rate, divided by the number of trials. That construction assumes the observed rate sits in the middle of a roughly symmetric range of plausible true rates, which holds when the rate is near a half and fails badly when it is near zero: a true rate cannot be negative, so the range of values still compatible with the data is squashed on one side and stretched on the other. Applied to eight events in 2,000 the symmetric formula duly hands back a lower bound below zero, which is not a cautious answer but an impossible one. The Wilson construction pulls the centre of the interval away from the observed rate by an amount that depends on how few events were seen, which is why the range quoted here is not centred on 0.4% and why its upper end sits further from the point estimate than its lower end does.

The factor of six is the interval read against itself. The before rate could be as low as 2.43% while the after rate is as high as 0.79%, or as high as 3.95% while the after rate is as low as 0.20%, and dividing the extremes each way is where threefold and twentyfold come from. What the data say without ambiguity is that the two intervals do not overlap. Non-overlap is a statement that something changed; it is not a measurement of how much.

def rate_report(hits: int, n: int, daily_conditional: int) -> str:
    """A point estimate on a small count is a number nobody should quote alone."""
    p = hits / n
    z2, denom = 3.8416, 1 + 3.8416 / n
    centre = (p + z2 / (2 * n)) / denom
    half = (1.96 / denom) * ((p * (1 - p) / n + z2 / (4 * n * n)) ** 0.5)
    lo, hi = max(0.0, centre - half), centre + half
    return (
        f"{hits}/{n} = {p:.2%}  95% CI [{lo:.2%}, {hi:.2%}]  "
        f"projected {lo * daily_conditional:,.0f}-{hi * daily_conditional:,.0f} per day"
    )

print(rate_report(8, 2000, 88_000))
# 8/2000 = 0.40%  95% CI [0.20%, 0.79%]  projected 176-696 per day

The projection is where an argument about closing a ticket actually lives. The service produces about 220,000 summaries a day, of which roughly 40% involve at least one internal-only note, so the constraint is live on about 88,000 requests a day — the conditional denominator from chapter 3, since scoring the rate against all 220,000 would divide by a population the rule never governed. A 0.4% rate on 88,000 requests is around 350 leaked summaries a day, and the interval's upper end is closer to 700. The change is avoiding something like 2,400 a day, which is a genuinely large improvement, and what it leaves behind is roughly 32,000 leaked summaries a quarter, each one an internal fraud note sitting in a field an agent is about to paste into an email.

Every figure there is the rate multiplied by that denominator, and the multiplications are worth doing because they are what turns a decimal into an object somebody can picture. Two in five of 220,000 daily summaries is the 88,000. The point estimate times that denominator is the roughly 350 a day. Either end of the interval times the same denominator gives the 176 to 696 the code prints, which is the same uncertainty as before, wearing units. The avoided volume comes from the difference between the arms rather than from either arm on its own: 3.1% down to 0.4% is about two and seven tenths of a point, and that much of 88,000 is the roughly 2,400 a day. The quarterly figure is the daily one times about ninety days. None of those steps is harder than a multiplication, and every one of them happened only because somebody declined to leave the number as a percentage.

Tightening the estimate is possible and worth pricing before anyone asks for it. Pinning 0.4% to within a tenth of a point needs on the order of 15,000 conditional cases per arm, which is 150,000 calls across two arms and five runs, and the frozen set would have to be rebuilt because 2,000 cases is all the labelled internal-only traffic this team has. The cheaper move in the other direction is worth knowing too: an arm that produces zero violations does not establish a zero rate, since with no events in n trials the 95% upper bound is about 3/n, so a clean run over 500 cases is consistent with a true rate of 0.6% and a clean run over 5,000 with 0.06%.

That last rule is worth carrying into any review where somebody reports a clean run, because zero events is the least informative outcome a small sample can produce. Zero is what a true rate of zero looks like, and it is equally what a rate small enough to hide in a sample of that size looks like, and nothing in the observation separates the two. The bound is the statement of how small "small enough to hide" was on this particular sample, so it scales with the sample rather than with the result. A clean run over five hundred cases and a clean run over five thousand are different claims despite producing identical output, and the difference between them is a factor of ten in what they rule out.

The residual does not arrive where the eval sampled#

The 2,000 cases were sampled from thirty days of production traffic, which makes them a good estimate of the violation rate on traffic that looks like the last thirty days. Running the same two arms over three additional slices produces a different picture. The red-team slices below are cases written by someone whose goal was to make the rule fail.

The natural slice behaves exactly as chapters 1 and 2 predict. The two red-team slices do not improve at all, and the fourth one is worse in a direction that is inside the run-to-run spread and is also what the mechanism predicts, which is a combination that deserves attention rather than a shrug. On these slices the layout change bought nothing, and the violation rate on inputs designed to cause violations is roughly one in two both before and after.

Why a natural sample cannot bound an adversarial rate is a fact about where the sample came from, and it is worth separating from the observation that the red-team numbers happen to be high. An eval estimates a rate over the distribution it drew from, which here is thirty days of production traffic, and what it reports is the share of that distribution producing a violation. An attacker is not drawing from that distribution. They choose their input, with the system's failure mode in mind if they can infer it, so the question that matters is not how often a randomly arriving note breaks the rule but whether any constructible note does — and a rate estimated from random draws is silent on that question at every sample size. The same fact explains both halves of the table. The layout change moved the mass of the natural distribution, and the attacker was never standing in the mass.

There is a second distributional fact hiding inside the natural slice. Of the 62 violations in the before arm, 44 came from four note templates, all of them exports from the same fraud-review tool, which formats its comments as a bulleted list with short declarative lines that read like summary material. The eval's estimate of 3.1% is therefore mostly an estimate of how often that tool's exports appear in traffic, and that fraction is a product decision made by whichever team owns the integration. A new fraud tool with a chattier export format moves the rate without a single token of the prompt changing.

More than two-thirds of the before arm's violations coming from four templates has two consequences, and neither of them is about prompting. The first is that the rate moves when that tool's share of traffic moves, with no code change anywhere and nothing to attribute the movement to, so a fall in the number is not on its own evidence that anything improved. The second is that the cheapest available reduction is not a prompt edit. It is a change to how one integration formats its comments, which is a conversation with the team that owns that integration, and it is the sort of fix that never surfaces in a prompt review because nothing in the review's frame reaches it.

Late placement buys proximity, not authority#

The reason the red-team slices did not move is worth being precise about, because it is the difference between a positional effect and a security property. Everything in this topic has treated position as a proxy for how strongly a span influences generation, and the tail wins because little competes with it at the moment the first output token is produced. Nothing in that mechanism distinguishes text the prompt's author wrote from text that arrived inside a retrieved document. Both are tokens in the same sequence, and the model weighs them by the same rules.

An attacker's payload lives in the retrieved note, which sits immediately before the tail block, so it is competing at almost the same distance the constraint enjoys. The fourth slice goes further and imitates the tail: a note ending with a line formatted the way the authored final block is formatted, asserting that the confidentiality requirement has been superseded for this ticket. Moving the real constraint into that position created a template for the imitation and taught the layout that instructions appear there, which is why that slice is the one that did not improve. Position confers proximity to the point of generation, and it does not confer authority over anything else in the window, because there is no channel in a flat token sequence that marks one span as trusted.

The fourth slice repays being taken apart, because it is the one where the remedy opened the door. Putting the real constraint in a formatted block immediately before the question makes that block the place instructions live and gives it a recognisable shape — which is the whole reason a human reviewer can find it. A note ending in a line of that same shape, asserting that confidentiality has been waived for this ticket, is then a plausible instruction in a plausible position, and constructing one costs an attacker nothing but the formatting. The model has no way to check the claim. Nothing in the sequence records who wrote which span, so the authored block and the imitation are both just text, both near the end, both formatted like rules. What distinguishes them to a person is provenance, and provenance is precisely the property a flat token sequence does not carry.

That is the sense in which the mechanism behind the improvement is a positional bias in a text predictor. It is a statistical regularity of how the model allocates influence over distance, measured on one model version, and it is not a property anyone can assert about inputs nobody has sent yet.