A provisional answer may only say what the real one cannot contradict

The requirement came down as a number: an answer visible within 400 milliseconds. The tool is legal research, the large model with retrieval takes about 3.0 seconds end to end, and a smaller model answering from the same retrieved passages comes back in 280 to 340. The proposed design was to render the small model's answer immediately and replace it with the large model's answer when it arrived, which does meet the number and was demonstrated working two days later.

In the demo, the question was whether a 2019 amendment extended the limitation period for a particular class of contract claim. The provisional answer appeared at 310 milliseconds and said the period had been extended to six years. The real answer appeared at 3.1 seconds and said the amendment applied only to claims arising after commencement, leaving this one at three years. Between those two events the associate running the demo had already read the first answer, because reading a sentence takes under a second, and in ordinary use would have already pasted it into a memo. The latency target was met on every request and the tool told a confident falsehood for 2.8 seconds on this one.

The line the provisional answer may not cross#

The rule that makes this pattern safe is narrow: the provisional response may contain only content that the large model cannot contradict. That is a much stronger constraint than "content that is usually right", and it rules out almost everything a small model is good at.

The dividing line is whether the statement has a truth value in the domain. "You asked about the limitation period for contract claims in England after the 2019 amendment, and I am reading four sources" cannot be wrong in a way the large model corrects, because it is a description of the request and of retrieval, both of which are already settled by the time the provisional renders. "The period is six years" can be wrong, and once it has been read, the correction arrives too late to matter.

Retrieval is the part of the pipeline that makes this workable, since on this system retrieval completes in 180 to 260 milliseconds, well inside the 400-millisecond budget. Showing the four source titles with their dates is real information, arrives fast, is never retracted, and gives the user something to evaluate while the answer is being written. It is also honest about what the system is doing, which the six-year claim was not.

Replacing without leaving a seam#

The second half of the pattern is the swap itself, and it fails in ways that are easy to reproduce and easy to miss in a demo. The provisional and the final must never be visible at the same time, in any order, for any duration. A progressive replacement — streaming the final answer into the same block while the provisional text is still partly present — produces a paragraph that is half one answer and half the other, and that composite is a sentence neither model wrote.

type Phase =
  | { kind: "provisional"; text: string }
  | { kind: "final"; text: string };

let phase: Phase | null = null;
const requestId = nextId();

// Both calls are in flight from t=0; whichever returns is handled here.
onProvisional(requestId, (text) => {
  if (phase?.kind === "final") return;          // final already landed: discard
  phase = { kind: "provisional", text };
  render(phase);
});

onFinal(requestId, (text) => {
  phase = { kind: "final", text };              // one assignment, one paint
  render(phase);
});

Three properties of that code are the whole design. The final path never checks what the provisional said, so there is no merge and no diff. The provisional path checks whether the final has landed and drops itself if so, which handles the case where the large model is unusually fast or the small one is unusually slow, and that case is not rare — on a system with a 300-millisecond provisional and a 3-second final, a cache hit on the large path returns in 400 milliseconds and the ordering inverts several times an hour. And the swap is a single assignment followed by a single render, so there is no interval in which both texts exist on screen.

Streaming the final answer complicates this, because the natural implementation replaces the provisional with an empty block that then fills token by token, which puts the user in front of a blank area for whatever the first-token time of the large call is. Holding the provisional until the final's first token arrives, then swapping, is the version that does not flicker, and it means the provisional is on screen for slightly longer than the pure latency numbers suggest.

Duplication is the failure that survives the bounding rule#

Once the provisional is restricted to content with no truth value, contradiction is designed out and a second failure moves to the front. The provisional says "You asked about the limitation period for contract claims in England after the 2019 amendment", and the large model, following its own prompt, opens with "You asked about the limitation period for contract claims in England after the 2019 amendment". The swap replaces one with a near-copy of itself, and the user sees a sentence flicker and reappear in slightly different words, which reads as a glitch rather than as an answer.

The fix is that the two prompts have to be written as one design. If the provisional owns the restatement and the source list, the final's prompt must be told those already exist and to begin with the substance. That coupling is worth writing down somewhere both prompts can be read together, because the failure only appears when they are changed independently — a prompt edit that re-adds a preamble to the final answer breaks the provisional's contract without touching the provisional's code.

The same coupling governs what happens on the surfaces where the swap is not clean. A provisional rendered above a streaming final, rather than in place of it, produces a page where the restatement stays visible while the answer arrives underneath, which avoids the flicker entirely and costs a permanent block of low-value text at the top of every answer. Whether that is better is a layout judgement rather than a latency one, and it is worth making explicitly instead of falling into it.

What the second call costs#

The provisional is a second model call on every request, which the latency framing tends to obscure. On this tool the small model runs about $0.0004 per call against the large path's $0.021, so the direct spend increase is under 2% and not the constraint. Concurrency is the constraint: every request now issues two calls instead of one against a rate limit shared with the rest of the product, and the same arithmetic that governed the fan-out applies here, with the additional wrinkle that the provisional's value collapses if it queues. A provisional that arrives at 900 milliseconds instead of 310 has missed the 400-millisecond requirement it exists to satisfy, and at that point it is pure cost.

It is also worth checking that the provisional is on the critical path at all. If retrieval finishes at 240 milliseconds and the source list renders then, the marginal value of the small model's restatement arriving 70 milliseconds later is small, and dropping the second call entirely may meet the requirement with one fewer moving part. That comparison is easy to run and easy to skip, because by the time anyone asks, the provisional has usually already been built.

Finding out whether it is hurting anyone#

Shipped with the bounded provisional above, the latency dashboard shows time to first useful content at 310 milliseconds p50 and 480 p95, which is an excellent-looking number and says nothing at all about whether the design is safe. Three instruments answer that question, and none of them is a latency metric.

The first is contradiction rate, measured offline. Run the provisional and the final on the same 2,000 historical queries, store both texts, and have a grader — a reviewer, or a model prompted to judge one pair at a time — decide whether the final contradicts the provisional. On a bounded provisional this rate should be structurally zero, and any non-zero result is a leak — a prompt that permitted a claim, or a small model that ignored the bound. On the original unbounded design this rate was 11%, which is the number the demo exposed by accident.

The second is the acted-before-swap rate. Instrument every user action that takes the answer out of the tool — copy, export, citation click, sending a message in the thread — with the timestamp relative to the swap. On this tool, median time from render to first copy was 1.4 seconds and the swap landed at 3.1, so roughly 60% of copies happened while the provisional was on screen. That single number is what turns "the provisional is sometimes wrong" into an estimate of how often a wrong claim left the building.

The third is post-swap correction awareness, which is the hardest and the one most teams skip. If the text under the user's eyes changes while they are reading it, do they notice? A session replay sample of fifty swaps — recordings that reconstruct the page exactly as it moved under the user's eyes — is enough to find out, and the usual answer is that a swap landing more than about a second after render, in a block the user has already begun reading, is frequently not noticed at all.

The label that gets proposed at this point — "still checking, this may update" above the provisional — does not move any of those three numbers. It transfers responsibility from the system to the user without changing what the user does, because people read answers and not disclaimers, and because the disclaimer is equally present on the 89% of requests where the provisional was fine, which is exactly the training that teaches users to ignore it. A disclaimer is a reasonable mitigation for a rare event. Here the event is every request, which makes it a design decision wearing a warning label.

Two neighbouring ideas that do not carry this risk#

Speculative decoding is the technique this pattern is most often confused with, and the confusion matters because speculative decoding has none of the same exposure. There, a small draft model proposes several tokens ahead and the large model verifies them in a single forward pass, accepting the ones it would have produced itself and discarding the rest. The output is token-for-token identical to what the large model would have generated alone, so there is nothing to retract and no user-visible provisional state; the speedup, typically 2x to 3x on accepting workloads, is invisible except as throughput. If speculative decoding is available on your deployment, it delivers part of what the provisional-answer pattern promises with no product decision attached.

Routing is the other neighbour. Sending easy queries to the small model and hard ones to the large model produces one answer per request, at the small model's latency for the queries it handles, and its failure mode is a wrong answer rather than a retracted one. That is a worse failure in isolation and a more familiar one to manage, because it is the ordinary quality problem that evaluation is built for.

On a legal-research tool the recommendation to put in front of product is the bounded provisional plus the source list, which meets the 400-millisecond requirement with content that has no truth value, and an explicit refusal to render a provisional holding at any latency. That is a smaller product than the one requested, and it is the version whose worst case is a slow answer rather than a confident wrong one.

Both of the remaining levers — sending the model the output you expect, and generating an answer before anyone asks for it — avoid the retraction problem entirely, because neither one shows the user anything provisional. What they spend instead is money, on every request where the guess was wrong.