Prompt accretion and ownership

Accretion is what a prompt does when every incident is closed by appending to it. Each addition is individually justified, references a real ticket, and makes the prompt better at the specific thing that had just gone wrong; the defect is a property of the sequence rather than of any commit in it, which is why it survives any amount of individual care.

Reading a prompt for accretion is a diagnostic skill worth having before any eval is run, because the symptoms are visible in the text and in the history, and they tell you where the deletions are. The subject here is a ticket-triage prompt of 6,200 tokens whose git log returns 47 commits by nine authors, three of whom have left the company, and where 41 of those commits add lines while two remove any — both of them reverts of the commit immediately before.

Three symptoms, all legible before an eval#

The first is the same subject addressed in several places.

# lines 1-4
You are a ticket triage assistant. Classify each ticket into exactly one
category from the schema below. Respond with a single JSON object and
nothing else.

# line 88, added 2025-04 after INC-2210
Always wrap your output in a fenced code block tagged json, so the parser
can find it.

# line 402, added 2025-11 after INC-3471
Return JSON. Include a `rationale` field explaining your choice so we can
audit misroutes.

Three instructions about output format, written by three people, that do not agree. The first forbids anything but the JSON object; the second requires a fence around it, which is something other than the object; the third adds a rationale field that is not in the response schema. The downstream parser strips fences defensively and drops unknown keys silently, so the disagreement has been invisible for eight months, and 1.8% of responses currently carry a rationale the audit tool was built for and never receives.

The reason nobody noticed is the parser, and the causal chain is worth following because it is the same chain in every prompt that drifts. A downstream component that strips fences defensively will accept the object whether or not a fence is there, so the conflict between line 3 and line 88 produces no error. A component that drops unknown keys silently will accept the object whether or not rationale is present, so the conflict between line 3 and line 402 produces no error either. No error means no incident, no incident means nobody reads the prompt, and nobody reading the prompt means the next instruction is appended rather than reconciled. Defensive parsing is good engineering that removes the only signal that would have made somebody look, and eight months is how long that takes to become three instructions.

Duplication written this way is a maintenance signal rather than a style problem. Somebody in April read the prompt, or did not, and added a format instruction rather than editing the one on line 3. Somebody in November did the same. Neither deleted the other's work, because deleting another team's incident fix is an act that requires standing, and nobody had any.

The second symptom is a reference to something that no longer exists. Two of the six few-shot examples show a priority field in their expected output. That field was removed from the response schema last quarter when priority moved into the routing service. The examples were not updated, because the person who changed the schema changed a TypeScript type, ran the type checker, saw it pass, and shipped — and the prompt is a string in a different file that no type checker reads.

This one has a bill attached. The response validator rejects objects with unknown keys, and 3.1% of triage responses still include priority, following the examples. Each rejection triggers one retry, so 3.1% of tickets cost two model calls instead of one and add about 1.4 seconds to their own latency. At 40,000 tickets a month that is 1,240 extra calls, which is small money and a persistent 3% of the error budget — the share of requests the service is permitted to fail or retry before it misses its reliability target — being spent on a field that has not existed since September.

Two multiplications sit in that paragraph and the second one is the expensive one. Three point one per cent of 40,000 tickets is 1,240 tickets a month that take two model calls instead of one, and at a per-ticket price of a fraction of a cent that is the small money. The error budget is the other figure and it does not scale with volume, because it is a share rather than a count: whatever the traffic, roughly three per cent of it is being spent on retries caused by two stale example outputs. An error budget is the allowance a service has for everything that can go wrong — a bad deploy, a dependency timeout, a genuine model failure — so a standing three per cent is not a cost that has been paid, it is capacity that has been removed before anything unexpected happens. Two examples nobody updated are holding a lien on the service's tolerance for its own future incidents.

The third symptom is the largest. Eleven clauses of the form "if the ticket mentions X, do Y" total 1,340 tokens, were added between June 2024 and January 2026, and each closed an incident where a category of ticket was routed wrongly. Appending one is the cheapest possible response to a misroute: it takes ten minutes, it is obviously related to the reported problem, and in review it looks additive, which means nobody has to argue that anything already in the file should go.

The asymmetry that drives that loop is the thing to fix, because it is not a failure of discipline. Adding a clause requires no evidence — the folk model says more instruction cannot hurt, and the incident is right there as justification. Removing one requires the frozen-set comparison, the per-case diff and the argument about rare populations that a few-shot deletion needs. When one direction costs ten minutes and the other costs two days, a file only moves one way, and that is true of any codebase with no owner: dead feature flags, a config nobody dares prune, a chain of if branches in a legacy service where each branch is somebody's production fix.

The folk model deserves to be taken seriously before it is dismissed, because almost everything about it is right. An added clause is a true statement about how tickets should be routed. It is short. It does not delete anything. The reviewer reading the pull request can check it against the incident and agree, and the check will pass, because the clause is correct. What the folk model omits is that a prompt is read whole, and instructions in it compete for the same attention that the crossover measurements were about. Two clauses about chargebacks stated in two vocabularies are not one instruction twice as strongly made; they are two weaker instructions, each carrying part of the signal, and the model has no way to know they were meant to be the same rule. The clause is correct in isolation and it is the wrong unit of correctness, which is exactly the kind of defect a per-commit review cannot see.

The eleven clauses, read as a set#

Each clause is defensible on its own, which is why each was merged. Laid out together with their dates and their triggers, they stop being eleven rules.

Four clauses written eighteen months apart say the same thing in four vocabularies, which is what the chargeback recall figure from the deletion work was quietly paying for. Two name products that no longer exist and matched zero tickets in the last ninety days, and nobody deleted them because nobody was looking at the prompt when the products were sunset.

The step from "four clauses say the same thing" to "recall on that class was lower" is the one the table cannot show, and it is the same mechanism the retrieval work measured, arriving from the other direction. A retrieved block diluted an answer because seven passages restated one policy and one passage stated an exception, so the prompt's own shape misrepresented the weight of the evidence. Here the restatements are hand-written, and the shape they produce is four partial descriptions of one category — one keyed on the word "chargeback", one on "disputed the charge", one on a statutory reference, one on the customer naming their bank. A ticket matching only one of the four presents the model with one clause that fits and three that do not, and there is nothing in the file saying the four are the same rule. Merged into a single clause listing all the trigger terms, the same ticket matches one rule that fits completely. No information was added and none was removed; the four descriptions were replaced by one, and that is where the recall came from.

Rows 7 to 9 are a different kind of problem and the most useful one to generalise from, because they are not classification. They name a destination queue, which the routing service already computes from the predicted class plus two fields it has on the ticket — the account's plan and the detected locale. Putting them in the prompt asked a language model to perform a lookup that a dictionary performs correctly every time. It performed it correctly on 96% of the relevant eval cases, and the 4% is not a tuning problem, since a deterministic mapping evaluated probabilistically has a floor above zero error by construction. Language detection is the clearest case: an off-the-shelf classifier gets German right at 99.4% on this traffic and costs nothing per call, and the prompt was doing it at 96% for 140 tokens on every request.

That last claim is the one to unpack, because "a deterministic mapping evaluated probabilistically has a floor above zero error" is a sentence a reader nods at and cannot rebuild. The routing service already holds the account's plan and the detected locale as fields, and the mapping from those fields to a queue is a lookup: the same inputs return the same output every time, because that is what a table does. Moving the rule into the prompt replaces the lookup with a chain of steps that each have to go right — the model has to find the plan in the ticket text or infer it, recall that clause 7 exists among ten others, apply it in preference to the classification instruction it is also being given, and emit the queue name. Each of those steps succeeds with high probability and none succeeds with certainty, and a chain of high probabilities is a lower probability. The model is also working from strictly less information than the service, since the plan is a field the service holds and a fact the ticket text may never mention. The 4% is therefore not a gap a better model or a better-worded clause closes. It is what it costs to have asked a probabilistic component to reproduce something that was already known exactly.

The language-detection row makes the same point with a number attached, and the comparison is not really 99.4% against 96%. It is that one of those figures can be improved by swapping a library and the other cannot be improved at all without also changing everything else the prompt does, because the 140 tokens doing the detection are sitting in the same context as the classification rules and competing with them. A dedicated classifier has one job and its errors stay inside that job. A clause in a shared prompt has one job and its tokens affect every other job in the file.

The test that generalises is whether the rule is decidable from fields the service already holds. If it is, the model is the wrong place for it, and moving it out removes tokens and an error source together. What belongs in the prompt is judgement about the ticket's text, which is the one thing the routing table cannot do.

What ownership and a budget actually mean here#

Ownership is a named team in CODEOWNERS — the file a code host reads to require a particular team's review on any change under a given path — covering the directory the prompt lives in, and a rule that changes to it are reviewed by that team rather than by whoever is on call for the incident. The point is having someone with standing to say no, and to say that a proposed clause duplicates one on line 220 and the right change is to edit line 220.

A budget makes the second direction cheap. The prompt gets a token allocation per component, in the shape the budget work established, and a check that fails the build when a component exceeds it. The effect is on how proposals are framed: with a budget, adding 180 tokens of new rule means either finding 180 tokens to delete or raising the allocation with a measurement attached, and the conversation about what the prompt is for happens at the moment the change is made rather than eighteen months later.

The mechanism is worth stating precisely, because a budget does not stop anybody adding anything. What it changes is who has to make the argument. Without one, an appended clause is a pull request that adds lines and passes review; the cost is real but it falls on the file rather than on the author, and there is nobody whose job it is to notice. With one, the same clause turns the build red, and the author now has two options that are both arguments: find tokens to remove in the same component, which forces them to read what is already there, or raise the allocation, which requires a measurement and a reviewer's agreement. Neither option is expensive. Both of them put a person in front of the existing contents of the file at the exact moment they were about to add to it without looking, and that is the whole intervention.

Provenance makes both possible. Each block carries who added it, which incident, and when it was last measured, which turns the audit from archaeology into a query.

import datetime, pytest
from prompts.triage import BLOCKS          # parsed from prompts/triage/system.md
from schemas.triage import RESPONSE_FIELDS # the live response schema

BUDGET = {
    "role_and_task": 400, "output_format": 250, "examples": 1400,
    "escalation": 700, "special_cases": 350, "tone": 150,
}   # 3,250 tokens total, agreed with the owning team

def test_each_block_is_within_budget():
    for name, block in BLOCKS.items():
        assert block.tokens <= BUDGET[name], (
            f"{name} is {block.tokens} tokens, budget {BUDGET[name]}. "
            f"Delete something or change BUDGET with an eval link."
        )

def test_no_block_names_a_field_the_schema_dropped():
    """The check that would have caught `priority` the day it was removed."""
    for name, block in BLOCKS.items():
        for field in block.referenced_field_names():   # regex over `backticked` idents
            assert field in RESPONSE_FIELDS, (
                f"{name} references `{field}`, absent from the response schema"
            )

@pytest.mark.parametrize("name,block", BLOCKS.items())
def test_block_has_provenance_and_recent_evidence(name, block):
    assert block.owner and block.incident_ref
    age = datetime.date.today() - block.last_measured
    assert age.days <= 180, f"{name} last measured {age.days} days ago"

The second test is the one that pays for itself immediately, because it converts a class of silent drift into a red build in the pull request that causes it. The third is deliberately annoying: a block whose evidence is more than six months old fails, and the fix is either to re-run the frozen set with and without it or to delete it. Most teams that adopt this discover that the majority of their prompt has never been measured at all, which is the finding rather than an obstacle to it.

Growth by design looks different in the file#

Shorter is not the objective, and a prompt cut on that principle will lose the material it was carrying for good reasons. The four surviving examples cost 1,200 tokens and hold a rare class together; the escalation block is 890 tokens of rules that a regulator cares about, it has never been measured against anything, and the correct next action is to measure it rather than to shrink it. The 3,980-token figure is a consequence of removing blocks that could not justify themselves, and a prompt that grew to 9,000 tokens through nine measured additions would be a healthy prompt.

The 3,980 is worth deriving rather than quoting, because doing so shows what the figure is and is not. The six steps removed 90, 145, 380, 410, 195 and 1,000 tokens, which add to the 2,220 in the total row, and 6,200 less 2,220 is 3,980. Notice what dominates: a single row, the trim from six examples to four, is 1,000 of the 2,220 — nearly half the saving — and it is the row that improved nothing. Notice equally what does not dominate: the row that produced the only quality gain, collapsing four chargeback clauses into one, removed 380 tokens. The two columns of that table are measuring different things and neither predicts the other, which is the argument against reading a clean-up as a token-reduction exercise. Had the goal been tokens, the last row alone would have delivered almost half of them in an afternoon, and the prompt would still contain three redundant chargeback clauses and a rule that asks a model to do a table lookup.

Growth by accretion and growth by design are distinguishable from the file alone, without knowing anything about the team. A prompt that grew by design has blocks that partition the problem — one place where output format is specified, one place where escalation is defined — and its history contains deletions, because each addition displaced something. A prompt that grew by accretion has the same subject addressed in several places by several vocabularies, has clauses whose triggers overlap, and has a history that is monotonically increasing. The second pattern is what nine authors and no owner produces, and it survives any amount of individual care, since the property being violated is one that no single commit can violate.

Accretion is also not cured by a clean-up. Six months after this one, the same file will be back above 5,000 tokens unless the review asymmetry has changed, because the incidents keep arriving and appending is still the fastest way to close one. The durable part is the owner, the budget test and the provenance requirement; the diff is the part that gets undone.

There is a structural answer to the eleven-clauses problem that goes further than merging them, which is to stop keeping conditional procedures in the always-loaded prompt at all. A rule that applies to 4% of tickets can live as a named, retrievable procedure the model loads when it recognises the trigger, which is the progressive-disclosure pattern described in Anthropic's Equipping agents for the real world with Agent Skills: the prompt carries a short index of what exists, and the full text of any one procedure arrives only once the model has decided it needs that procedure. It carries exactly the trade this topic has been measuring: fewer tokens on every request, and a new way to fail when the model does not load the procedure it needed. The instrumentation from the policy-lookup work — hit rate, contradiction count, per-case deltas — is what tells you whether the trade came out ahead on your traffic.

A prompt with an owner, a budget and provenance on every block is one where selection happens continuously instead of in a quarterly archaeology session. That is the whole point of the three mechanisms: they make removal as cheap as addition, which is the asymmetry that produced the 6,200 tokens in the first place. None of them requires anybody to be more careful than they already were.

The inventory that starts it is four columns and can be filled in for most production prompts in an afternoon: each block, its token count, the reason it exists with a link to the incident or the eval that justified it, and the date somebody last checked whether it still earns those tokens. Rows where the third column is empty are the ones to attack, and rows where the fourth is empty are most of them. Take the largest block with no evidence behind it, run the frozen set with and without it on identical settings, print the per-case diff rather than the mean, and read the regressions. Either you delete something and can say why, or you keep it and it now has a date in the fourth column. Selection is that loop, run on one block at a time, and the token count falls out of it rather than being the thing you were aiming at.