Auditing an inherited prompt

Inheriting a prompt usually means inheriting three symptoms at once: a latency number somebody wants lower, a model bill rising faster than traffic, and a quality complaint specific enough to have reached support. The prompt is the visible artefact in the middle of all three, it is editable without a migration, and it is long — so the proposal that arrives with the handover is a word-count target. Cut it to 400 words this sprint: specific, measurable, shippable.

It is the wrong first move for two independent reasons, and separating them matters because they lead to different work. The first is that nobody has yet shown the prompt is causing any of the three symptoms, and two of them decompose into numbers you can read in ten minutes. The second is that hitting 400 words means deleting roughly 1,900 chosen by a criterion — length — that is uncorrelated with which paragraph is load-bearing.

Diagnose before you cut#

A bill is attributed by division, and division needs both numbers. What the prompt costs on its own says nothing about whether it is worth cutting, since the same daily figure is either the whole problem or a rounding error depending on what sits beside it. The proposal that arrives with the handover is a numerator with no denominator.

Start with the two numbers that came with a unit. A 2,300-word prompt is roughly 3,100 tokens. If it is a stable prefix — byte-identical from the first token on every call — a provider serves it from cache at about a tenth of the input rate, so the honest way to see whether it is your problem is to put every component of a request side by side. For a service doing 500,000 model calls a day on Claude Opus 5's list rates, with an average conversation tail of 4,200 tokens behind the cache breakpoint and 600 output tokens per call:

Every row of that table is one multiplication. The prompt sends 3,100 tokens on each of 500,000 calls, which is fifteen hundred and fifty million tokens a day. A rate quoted per MTok is a rate per million tokens, so that volume is fifteen hundred and fifty of the units being priced, and at about half a dollar each they come to the $775 in the first row. The same two steps give $10,500 for the conversation tail and $7,500 for the output, and the three add to the $18,775 total. One conversion sits under all of it: 2,300 words of prose came to 3,100 tokens, a little over four tokens for every three words, which is the exchange rate between the unit people write targets in and the unit providers bill in.

The system prompt is 4% of the bill. Cutting it by 87% removes about $645 a day from an $18,775 day, or 3.4%, and a 40% quarter-over-quarter rise is sitting almost entirely in the two rows underneath — a growing conversation tail, which lengthens with every turn because every turn is re-sent, and longer outputs. If the prompt is not cached, the first row is $7,750 instead and the prompt is suddenly 30% of the bill; that changes the priority completely, and the difference between those two worlds is one cache_read_input_tokens value in a response. Find out which world you are in before you spend a sprint.

Finding out is a field rather than an experiment. Whether a prefix is served from cache depends on it arriving byte-identical call after call — the subject of the chapter on prompt caching and prefix stability earlier in this topic — and none of that is visible in the prompt text. It is reported back instead. A response whose cache_read_input_tokens is close to the prompt's own token count was served that prefix from cache, and one reporting nothing there paid full rate on all of them.

Latency decomposes the same way. Prefill on a cached 3,100-token prefix is on the order of tens of milliseconds; generating 600 output tokens is the overwhelming majority of a 3.1-second p95, and the tail is usually dominated by long generations rather than long prompts. A prompt cut can move latency, and the mechanism is that a shorter prompt sometimes produces shorter answers, which is a behavioural effect rather than a prefill one and is worth naming as such so that you measure output length rather than prompt length.

The gap between prefill and generation comes from the loop that produces the answer. A model call reads the prompt once before the first piece of the answer appears, then runs the model again for every further piece, which is the loop topic 1.0 sets out. Prompt tokens add work once; output tokens multiply it, since 600 of them is six hundred passes each waiting on the one before.

A complaint about the assistant asking too many questions before answering is the one symptom the prompt is plausibly responsible for, and it is usually the one nobody proposes investigating. It is also the cheapest to check, because a complaint that specific normally maps to a sentence.

The reason it maps to a sentence is that the prompt is the part of the request that does not vary. Conversations differ, customers differ, whatever gets retrieved differs; the standing instructions go out unchanged on every call. A symptom appearing across unrelated conversations therefore has one shared cause available to it, where cost and latency scale with whatever those conversations happen to contain and point back at the rows that grew.

The first three things to remove#

Read the prompt with a notebook and mark every rule with where it came from and whether anything downstream disagrees with it. Three categories come out first, ahead of anything chosen for length.

The contradiction is first because it is the only category that is actively degrading behaviour on every request. A prompt that says one thing in section 3 and the opposite in section 7 looks like this:

§3  Always ask a clarifying question before answering, to make sure you
    have understood the request.
...
§7  Never ask more than one question; give the answer directly and offer
    to refine it afterwards.

Reduced to what each demands of a single turn, those rules are Always ask a clarifying question before answering. and Never ask more than one question; give the answer directly. Reducing them that way is the move that finds contradictions, since neither line looks wrong beside the complaint it was written to fix. What is there is not the formal clash of a rule with its negation: section 3 puts a question before the answer and section 7 puts the answer first, so no single turn satisfies both.

Both were added by different people solving different complaints, months apart, and neither read the whole document. The model is now resolving that conflict per request, which is why the behaviour is inconsistent rather than uniformly bad, and why support complaints describe an assistant that "sometimes" interrogates people. Resolving it means picking one and deleting the other outright, rather than softening both into something that mentions clarifying questions twice. Then search the rest of the document for the same pattern, because a prompt with one contradiction reliably has three: dozens of commits by several people is a merge process with no reviewer holding the whole text in their head. Both the diagnosis and the fix are free of measurement risk, which is unusual and worth using — you are removing a sentence that no path in your product wants to be true.

Nothing in the request stops at the contradiction and reports it. A model call scores continuations of the text it was handed, which is the loop chapter 1.0 describes; no step checks the instruction set for consistency, and no call comes back saying two of its instructions cannot both be honoured. What comes back is a turn consistent with one reading of the document, and which reading wins depends on where each rule sits, how each is worded, what else the conversation contains — and on a picking step that is allowed to be random.

That dependence is what puts the contradiction first, ahead of the categories that save more tokens. Every later step in this review measures behaviour: remove a rule, replay the traffic where it applied, compare. The comparison assigns the change to the change you made, and it holds only while nothing else about the system is moving. A live contradiction is something else moving, on every request, so a slice replayed against a prompt that still contains one produces differences nobody can attribute to the rule under test. The order of the three categories is a dependency rather than a ranking by value.

Softening the two clarifying-question rules instead of deleting one is the tempting alternative, and it appeals because each line has an author and a ticket behind it, so deleting one feels like discarding somebody's requirement. What it does is convert a contradiction two people can find by reading into one that shows up only in behaviour, with the model still deciding per request how much clarifying to do.

The obsolete instruction is second. If the user asks about the mobile app, say that it is coming soon and offer to take their email is not merely wasted tokens once the mobile app has shipped: it is a rule that fires on real traffic and produces a wrong answer every time it does, and the reason nobody notices is that it fires on maybe 0.3% of conversations, which is invisible in an aggregate quality score and extremely visible to the two hundred people a week who hit it. The way to find this class is to read every conditional rule and ask when it last matched reality, and the way to keep finding them is a date and an owner next to each one.

Two things put that category second rather than first. Its damage is bounded by the traffic matching its condition, where a contradiction's is not, and it blocks no later measurement. It is also the only one settled without touching the model, since its truth condition is a fact about your product: somebody confirms the mobile app shipped, and the rule is dead by inspection.

The distance between 0.3% of conversations and two hundred people a week is one set of events counted two ways. An aggregate divides by all traffic, so a rule firing rarely moves that number slightly whatever it does to the conversations it touches. A support queue performs no division; it accumulates the people affected for as long as the rule is there. The inbox is the reading that describes what the rule did.

Prose that restates a model default is third, and it is third rather than first because removing it usually changes nothing at all — which is precisely what makes it safe and low-value. You are a helpful assistant. Be accurate, thorough, and clear. Provide complete answers. A current instruction-tuned model does these things unprompted, so the paragraph costs tokens and buys a behaviour you already had. The nuance is that "restates a default" is a claim to be tested rather than assumed, and the test is the same one as for everything else below: remove it, run the slice where it would bind, and look at whether anything moved.

The claim being tested there is a claim about a particular model, and that condition is the whole of what makes the third category safe. The behaviour comes from the model's own tuning, so the paragraph is redundant for as long as that tuning holds, and a model swap is the event that ends it. A paragraph measuring as dead today can be the only thing producing a behaviour against the model that replaces it.

There is a matching list of things to leave alone, and it is the half of the review that a length target destroys. Anything that only you know stays: what the product is, who the audience is, what the quality bar looks like, which of your systems the assistant is talking to, and the reason behind each constraint — a rule with its reason attached survives contact with a case its author did not anticipate, and the same rule stripped to an imperative does not. Exact wording stays where exactness is the point, which covers regulatory sentences, the output schema, and any procedure with one safe ordering. Examples stay when they pin down a format the model gets wrong by default, though they are worth re-reading for interpolated values that turned a stable block into a per-request one. Working duplication is not urgent: two sections that say compatible things about tone are a tidying preference, while two sections that disagree are the first item on the list above. The distinction that makes the review tractable is between text that constrains behaviour, which you can test by removing, and text that supplies context the model has no other way to obtain, which you cannot.

What is conspicuously absent from that list is the longest section. Length is not evidence of anything, and the paragraph most likely to be load-bearing in a mature prompt is often the long one, because it is long for the same reason it is load-bearing: someone had to spell out a case the model gets wrong by default.

Length misleads in both directions, which leaves neither party to the argument on solid ground. The engineer proposing a target reads length as fat, and the engineer defending the document reads it as necessity, on the ground that every line was added for a reason — but a line's reason was a state of the world at the time, and it does not travel with the line unless somebody wrote it down alongside. What the shape does say points the wrong way for a cut: rules get long by enumerating cases, and cases get enumerated where the default behaviour was wrong.

The division between text that constrains behaviour and text that supplies context survives removal on one side only, which is what makes it the tractable one. Constraint text leaves the model able to produce the behaviour anyway, so removing it and replaying the traffic answers the question directly. A fact the model cannot obtain elsewhere leaves nothing to test against, since its absence comes back as a fluent answer assembled from whatever is plausible — the failure topic 1.0 closes on — rather than as a blank.

Validating a cut rule by rule#

The reason a 400-word target is dangerous is not that shorter prompts are worse. It is that a dropped rule does not fail visibly. It produces a slightly wrong format, a missing disclaimer, or a dropped edge case on the slice of traffic where it applied, and every one of those slices is small enough to vanish in an aggregate. Ship the cut, watch the dashboards for a week, and the dashboards will say fine, because a rule that binds on 2% of conversations can go from 98% correct to 40% correct and move the overall number by about one percent.

That last figure is worth deriving rather than taking on trust, because the procedure rests on it. An aggregate score is an average over all traffic, and an average splits into the slice and everything else. The slice is 2% of conversations and its own score falls from 98% correct to 40% correct, a drop of fifty-eight points. The other ninety-eight per cent never met the rule, so its score is unchanged and contributes nothing to the movement — which is what makes it legitimate to multiply the slice's drop by the slice's share and stop there. Fifty-eight points on two hundredths of the traffic is a little over one point on the overall number.

So validate per rule. For each rule you intend to remove, the question is not "is the prompt still good" but "what does traffic look like where this rule binds, and did that change". That gives a concrete procedure:

# One slice per removed rule: the conversations where that rule applies.
SLICES = {
    "reply_in_customer_language": q("lang(customer_msg) != 'en'"),   # ~1,900/wk
    "eur50_escalation":            q("intent = 'refund' AND amount > 50"),
    "gdpr_deletion_disclaimer":    q("intent = 'delete_my_data'"),   # ~40/wk
}

for name, slice_query in SLICES.items():
    convos = sample(slice_query, n=200)
    before = replay(convos, prompt=OLD)
    after  = replay(convos, prompt=NEW)
    report(name, grade(before), grade(after), diff_rate(before, after))

Two things about this are worth stating plainly. The slice is defined by where the rule binds, not by a random sample, which is the entire point: a random 200 conversations contains perhaps four where a language rule matters, and four is not a measurement. And the smallest slices carry the highest stakes — a data deletion disclaimer that fires forty times a week would never register in any aggregate, and it is the one that generates a regulator's letter rather than a support ticket.

What makes the procedure possible at all is that a rule's binding condition can be written as a query over conversations you already store. The three entries in that SLICES map are predicates rather than names: a language rule binds where the customer's message is not in English, a refund rule where the intent is a refund above a threshold, the GDPR deletion disclaimer where the intent is a deletion request. Each is expressed in fields the conversation store already holds. A rule whose condition cannot be written that way — one binding when the customer sounds frustrated — cannot be sliced at all, which is a property of the rule rather than a gap in the tooling.

The random-sample remark is the same weighting seen from the other side. At the two per cent prevalence used above, two hundred randomly drawn conversations contain about four that the rule touches, and four is not a measurement of a rate: one conversation landing the other way moves the measured rate by a quarter, so the smallest change the sample can report is larger than any change worth catching.

Replay-and-diff is the cheap first pass for everything else. Run last week's real conversations through both prompts, diff the outputs, and read only the ones that differ. On a well-behaved cut the diff rate is a few percent and the diffs are all in the slices you predicted; a diff rate of 30% on a change you expected to be cosmetic means the prompt was doing more than you thought, and that result is worth more than the tokens you were going to save.

A null result is the reading to be most careful with. "Nothing moved" is a statement about the resolution of whatever did the looking, and for an aggregate that resolution is the number calculated above. The same removal has room to show a change of any size on the slice where the rule binds, so a line surviving an aggregate test has not yet been shown to be dead. The paragraph everybody had labelled restated defaults, below, is what that looks like when the labelling was wrong.

Where an aggressive cut is genuinely dangerous#

The per-rule procedure also tells you where it cannot be run, and those are the codebases in which the aggressive version of this work should not be attempted at all. If no rule has provenance — no ticket, no date, no owner — you cannot construct the slices, so every removal is untested by construction and the only honest options are to leave the prompt alone or to reconstruct the provenance first. If the prompt carries compliance text, the cost of a wrong cut is not a quality score, and the review needs somebody who can say which sentences are load-bearing in a legal sense.

Provenance matters in the first of those cases for a mechanical reason. A slice query is a restatement of a rule's binding condition, and a rule arriving as a bare imperative has no recorded condition to restate. Guessing one produces a slice that reports the removal as safe while never having looked at the conversations that mattered, which is worse than running no test at all: it produces a number somebody will quote.

Multi-tenancy is the case that catches careful teams. A rule that looks dead in your aggregate traffic may be the only thing holding one large customer's integration together, and the slice that would have shown it is a tenant filter nobody thought to apply.

And if the prompt has never been cut before, expect the first pass to find contradictions rather than fat, which will take a week and give you most of the quality win. That is the ordinary state of a document several people have edited under deadline.

Each of the three removals needed a fact the prompt itself does not contain: whether another rule contradicts this one, which is a fact about the whole document and so survives nine editors who each read a paragraph; whether the state of the world it describes still holds, which is a fact with a date on it; and whether it still changes behaviour, which is a fact about a slice of traffic and about the model running when somebody last checked.

The durable fix is not a shorter document but a document whose rules can be audited. Keep a ledger beside the prompt with one row per rule: the sentence, who added it, the date, the incident or ticket that motivated it, the slice of traffic it binds on, and when its effect was last verified. Adding a rule then costs a row, removing one costs a query, and the next engineer to inherit this prompt gets to start where you finished rather than reconstructing fourteen months of intent from a commit log. Start that ledger with the rules you removed this week, and add the ones you moved rather than deleted — the date and the customer's name, which were never rules at all, and which belong in the turn where they can vary without costing anything.