Examples that contradict constraints

Few-shot prompting means pasting a handful of worked input-and-output pairs into the prompt so the model has something to imitate. The pairs are ordinary input tokens; nothing about them is marked as illustrative, and nothing tells the model that the prose above them outranks them. What makes this worth a chapter of its own is that the resulting failure is invisible in review: a diff that adds three good examples to a prompt adds text and removes nothing, which is the shape of change that gets approved in four minutes.

The property has two edges, and only the first is widely known. A demonstration that contradicts a rule degrades compliance with that rule substantially. And a demonstration that contradicts nothing still specifies only the values it happens to contain, which is where invented enum members come from — a model shown one example with urgency: "high" has been told nothing at all about whether critical is legal.

An instance outranks a claim about instances#

Put a rule and a demonstration in one prompt and let them say different things about what a correct output looks like. The rule says the space of legal outputs excludes some property. The examples say: here are three outputs, drawn from the space of legal outputs, and two of them have that property. Those cannot both be true, and the model resolves the contradiction toward the examples with high reliability.

That resolution runs the wrong way round for most readers, and the expectation it violates is a reasonable one worth naming before it is dismantled. In every other specification an engineer works with, prose is normative and examples are illustrative. A standard says what a conforming implementation must do and then offers a sample, and the sample is explicitly non-binding, so where the two disagree the text governs. A linter reads the rule and not the code sample in the comment above it. A schema rejects a document; the example in the README does not. Somebody who has spent a decade with artefacts built that way will read a prompt the same way and conclude that a rule stated once outranks an example shown three times, because everywhere else it does. Nothing in a prompt implements that convention, and nothing in a model was built to honour it.

The reason is structural. The model is predicting the next tokens of a document, and the examples are the part of that document that most closely resembles the region it is about to write — same shape, same register, same position relative to the input. A sentence of policy is a claim about a set; an example is a member of the set. When the two disagree, the evidence closest in form to the thing being produced dominates, and the rule reads less like a law that was broken and more like a description that turned out to be slightly inaccurate about the data it accompanies.

That paragraph is the answer in compressed form, and compressed is not the same as explained. It asserts that the closest evidence dominates without saying what makes one piece of text closer than another, or why closeness should decide anything at all. A reader who wants to predict the next case rather than recite this one needs the chain underneath it, and the chain has five links, each of which can be checked on its own.

The first link is what actually arrives at the model. A language model is a scoring function: text goes in, and out comes a score for every fragment of text that could come next, after which a separate step of ordinary code picks one of those candidates, appends it, and calls the function again. What matters here is that the thing the function receives is a single string. Your constraints, your examples and the user's input are concatenated into one piece of text, and no part of that text carries a flag saying this part binds or this part illustrates. The role labels an API exposes — system, user, assistant — are markers written into the same string in a format the model was trained to recognise, not separate channels with separate authority. There is no field for normative and no field for illustrative, because there is only one field.

The second link is what the scoring function was shaped to do, which is continue a document plausibly. That is worth taking literally, because the strongest regularity in almost any document is local. A pattern established a few hundred tokens ago is overwhelmingly likely to continue: the fourth row of a table looks like the first three, the third bullet looks like the first two, the second paragraph of a letter sits in the register of the first. A function shaped by an enormous quantity of ordinary documents carries that regularity heavily, because ordinary documents have it.

The third link is where the examples sit. They are immediately above the point where the next output is due, in the same shape, in the same register, standing in the same relation to an input as the output the model is about to produce. On the next call the model is not being asked to satisfy a specification. It is being asked to write the next member of a series whose previous members are visible a few lines up.

The fourth link is that a prose rule and a demonstration both move the scores, and they move them by different routes. The rule raises the score of continuations consistent with an English sentence about what is allowed, and that route exists at all because of instruction tuning — the training stage in which a model is taught to follow written directions rather than simply continue the text it is given. Without that stage there is no reason a sentence of policy would govern anything. The demonstration raises the score of the literal strings that appear a short distance above in the same syntactic slot, and that route needs no special training stage to explain it, because it falls straight out of predicting the next token of a document.

The fifth link is the resolution, and it is the one that explains the shape of the damage rather than merely its existence. On an input resembling one of the demonstrations, the second route is operating on evidence in the same shape and the same position as the thing being written, and it dominates. On an input resembling none of them, the second route has little to say and the rule governs unopposed. Compliance therefore neither collapses to nothing nor holds at its old level. It falls on exactly the cases the examples resemble, and a measured compliance figure after a bad example is largely a report of how much of your traffic looks like that example.

Said plainly, now that the working is done: the rule does not reach the model as a law that has been broken. It reaches the model as a description sitting above some data, and the data disagrees with it. In a document, when a description and the data under it disagree, the ordinary conclusion is that the description was a little loose.

This is not a defect that will be patched. It is the same mechanism that makes few-shot prompting work at all, and it was the original interface. When the GPT-3 paper landed in 2020 under the title Language Models are Few-Shot Learners, examples were essentially the only way to specify a task, because instruction-following was weak enough that "classify the sentiment of this review" often produced a discussion of sentiment classification rather than a label. Instruction tuning, the fourth link above, changed that and made prose specifications viable, which is why prompts today are mostly written rather than demonstrated. What did not change is the relative weight: examples remain the highest-fidelity evidence of what an output is, and a prose rule is a lower-fidelity signal about the same thing.

The counterfactual is the quickest way to see that the failure and the feature are one property. Suppose a demonstration did not outweigh the prose around it. Then showing a model three worked cases would teach it nothing the accompanying sentences had not already said, few-shot prompting would be a way of spending tokens on redundancy, and nobody would have bothered to name the technique. It works because a demonstration carries more weight than a description of the same thing. A prompt that gets that weighting for free on the cases it thought about gets it for free on the cases it did not.

The operational consequence is a review rule with no exceptions. Every example in a prompt is a specification, so every example must be checked against every constraint, and the check is mechanical enough to automate:

FORBIDDEN = {"Datavail", "Northwind", "Corva", "Blackline Systems"}

def test_examples_obey_constraints():
    for name, example in FEW_SHOT_EXAMPLES.items():
        leaked = {c for c in FORBIDDEN if c.lower() in example.output.lower()}
        assert not leaked, f"{name} demonstrates a competitor mention: {leaked}"
        assert example.output.startswith("{"), f"{name} has a preamble"
        CallSummary.model_validate_json(example.output)   # shape matches schema

Each of those three assertions pairs one example against one class of constraint, and the correspondence is worth reading off directly. The leaked set is a content constraint: it takes the forbidden vocabulary the prompt claims to exclude and searches every demonstration's output for it, lowercased on both sides so that a competitor named mid-sentence is caught as readily as one at the start of a line. The startswith("{") assertion is a format constraint, because the prompt says a reply is a bare JSON object with no preamble, and an example opening with anything else is a demonstration of the preamble the prompt forbids. The model_validate_json call is a schema constraint, and it is the cheapest of the three to write, because the Pydantic model from chapter 1 already exists to validate replies at runtime — pointing it at the examples reuses one specification against two populations of text.

Nine lines, running in seconds with no model calls, and it catches a 35-point compliance regression at commit time. The test is worth having even in prompts you are confident about, because examples get edited by people looking at prose quality and constraints get added by people looking at compliance, and those are usually different pull requests months apart.

The absence of model calls is what makes it cheap enough to run on every commit, and the reason it needs none is worth stating rather than assuming. Both sides of every comparison in that test are static text in your repository. The examples are a file. The forbidden list, the no-preamble rule and the schema are also files. Nothing there asks what the model would do with the prompt; it asks whether two artefacts you already shipped agree with each other, which is a question decidable by reading them. A check that needs completions is slow, costs money on every run, and inherits the run-to-run variation chapter 5 measures, so it ends up gated behind a nightly job. This one is an ordinary unit test, and nobody has to weigh whether it is worth running.

What one example says, and everything it leaves open#

The second edge shows up in a debate that recurs on every team. Two prompts extract five fields from a support email. Prompt A describes the output in prose:

Return JSON with keys id, urgency, product, summary and next_action.
urgency is one of: low, medium, high.

Prompt B gives no description at all — one real email, followed by the exact JSON it should produce:

Email:
From: r.alvez@meridian.example
Subject: card reader offline in store 12 since this morning

Our card reader in store 12 has been down since 07:40. We are taking
cash only. Ticket 88214 is already open with your hardware team.

Output:
{"id": "88214", "urgency": "high", "product": "card-reader",
 "summary": "Store 12 card reader offline since 07:40; cash only.",
 "next_action": "escalate_hardware"}

The instinct is that B is better, because a concrete example is clearer than a description of a format. B is better at several things, and it is the one that will emit values your schema does not allow.

Prompt B specifies more of the things that are hard to write down and none of the things that are easy. An example is an existence proof: it establishes that high is a legal value of urgency, and it says nothing whatever about critical, urgent, p1 or sev-2, all of which are things support emails contain and all of which the model will produce on inputs where the example gave it no cover. The failures cluster exactly where you would expect — on the emails least like the example — which is also where they are hardest to catch in a spot check, because your spot checks tend to use inputs that resemble the example you wrote.

"Existence proof" is carrying real weight in that paragraph, and unpacking it is the closest this chapter comes to algebra. A demonstration and a closure constraint are claims of different logical shapes. A demonstration is an existential claim: there exists a legal output, and here it is — this particular object, with high sitting in the urgency field, is a correct answer. A closure constraint is a universal claim, and a negative one: no legal output carries any value in urgency other than these three.

Now count what a collection of existential claims can entail. One example establishes that one output is legal. Two establish that two are. A hundred establish that a hundred are, and the set of outputs the hundred have said nothing about is still every output that is not among them. At no point does adding another instance convert the collection into a statement about the outputs it does not contain, because a universal negative is a claim about the whole space while each existential claim is a claim about one point in it. The gap between the two shapes does not narrow as examples accumulate. It is the same gap after a hundred demonstrations as after one, which is why no quantity of them ever manages to say "and nothing outside these".

An invented enum member is therefore not a lapse in the model's attention. Shown one object with high in it, the model has evidence that high is legal and no evidence in either direction about anything else, so the field is undetermined and the plausible continuation wins. On an email that reads like a genuine emergency, the plausible continuation is whatever an emergency is usually called, and nothing in Prompt B competes with it. Prompt A does compete with it, using a single line that names the whole set — which is the shape of claim that can do the job, and which no demonstration has.

Prompt A has the opposite profile to Prompt B's, specifying the easy things and leaving the hard ones open. It closes the value space for urgency and leaves the shape of summary entirely to the model, so you get legal enum members attached to summaries that are three sentences on one email and a fragment on the next. Both prompts underperform a prompt that carries both, and the shipping answer is the boring one: state the value space in prose, demonstrate the shape with an example, and make certain the example only contains values the prose allows. If your example uses urgency: "high", add a second example using low, so that the demonstrated set and the stated set are the same set. The cost of that second example is about forty tokens.

If you are forced to ship one of the two as they stand, ship A, and the reason is about which failures you can see. Prompt A's failures are variation inside a legal envelope: every reply parses, every urgency is one of three values, and the summaries are inconsistent in ways a human reviewer will notice within a week and can fix with one added sentence. Prompt B's failures arrive as valid JSON carrying "urgency": "critical", which passes a shape check, fails a schema check if you happen to have one, and silently corrupts any downstream routing that switches on the three known values. A prompt whose failure mode is visible inconsistency is a better thing to have in production than one whose failure mode is confident illegality, and that ordering holds well beyond this example.

Placement is worth one more note, because it is the fix people try first. Examples sit below the rules in almost every prompt, for the ordinary reason that they read as illustrations of what came before, and that ordering does nothing to protect the rules — a contradiction survives a three-paragraph gap and would survive thirty. Moving a constraint closer to the examples, or repeating it after them, buys back some compliance in practice and buys back much less than deleting the contradiction. Treat restating a rule near the examples as a hedge on a prompt you have already made consistent, never as the repair.

Why distance should matter so little follows from the chain above, and it is the part people find least intuitive. The rule reaches the scores through the model having been trained to weight written directions, and that route is largely indifferent to whether the direction sits three paragraphs up or thirty. The demonstration reaches the scores by being the nearest thing in the document that looks like the output about to be written, and moving a sentence of policy nearer does not make that sentence look any more like an output. Proximity is not the lever it appears to be, because the two routes are not competing on proximity in the first place. Restating the rule after the examples does buy back a little, which is a different lever — a rule stated twice is stated twice — and neither lever touches the contradiction itself.

Every example being a specification is also the sharp answer to the triage pull request in chapter 2, which proposed two more examples against a prompt with no constraints. Each added example is one more artefact that must agree with every rule you have not written yet, and one more place for an unintended pattern to be demonstrated. If six examples happen to include four billing tickets, you have anchored the prior toward billing without meaning to; on one classification prompt where three of four examples used the same category, that category's share of predictions ran at 68% against a base rate of 31% in the eval set. Examples are the most powerful slot and the one that does the most work you did not ask for.

Fine-tuning, and rules that still do work#

Examples do not update anything. Nothing here is about the model learning your task in a way that persists past the request; the demonstrations are input tokens, they influence one completion, and they are re-sent and re-billed on every call. That distinction matters when someone proposes replacing a well-specified prompt with fine-tuning, meaning a further round of training that adjusts the model's own weights on a corpus of your examples rather than sending those examples on every call. Fine-tuning moves the demonstration into the weights, changes the economics at high volume, and inherits the same contradiction problem in a much less visible place, since a training set that contradicts a constraint is far harder to audit than three examples in a file you can read.

The chapter's rule also does not say that examples override rules always, and overstating it will lead you to the wrong fix. What holds reliably is that a demonstration contradicting a rule degrades compliance with that rule substantially, and the direction is consistent across models even when the magnitude is not. What does not hold is the inverse claim that a rule is worthless in the presence of examples. Remove the competitor constraint entirely from that prompt and compliance falls further than 61%, because the rule is still doing work on the two-thirds of cases where the examples do not apply. The rule and the demonstrations are both operative; when they conflict, the demonstration wins the cases it resembles.

That last claim needs its conditions stated, because it is the one sentence here that is a prediction rather than a measurement. Nobody in the account-summary case ran the third configuration — the examples in, the constraint deleted — so "compliance falls further" is what the five links predict and not a number anyone recorded. The prediction rests on one quantity, which the sentence above puts at roughly a third: the share of records the three examples resemble closely enough for the demonstration route to dominate. Where that share is small, the constraint is still governing most of the set and deleting it costs a great deal. Where it approaches the whole set, the constraint has already lost everywhere it could lose, and deleting it costs almost nothing.

A reviewer who hears the claim should therefore ask for the third run rather than accept the reasoning, and the run is cheap: the same fixed 250 records, the same decoding, one prompt carrying the constraint and one without it, both carrying the offending examples. What that comparison isolates is the residual value of a rule that is being contradicted — a quantity worth having a number for before anybody argues that a contradicted rule may as well come out.

So the review question is not whether a prompt says the right things. It is whether everything in it says the same thing — and because a demonstration and a sentence can disagree while both being individually correct, that question cannot be answered by reading either one. It is answered by a test that reads both against each other, and by a score on a fixed set of cases run before and after, which is the only instrument that would have caught 96% becoming 61%.

Which raises the case this chapter has been quietly assuming away. Those numbers were a disagreement between a rule and a demonstration, and you fix them by editing the demonstration. There is a second class of failure where both parties to the disagreement are rules you wrote deliberately, both are correct, and there is no example to blame.