Migration cost is revalidation, not rewriting API calls

The number given in the planning meeting was two weeks. It came from a senior engineer who had read the candidate provider's API reference the night before, noticed that the request and response shapes were close to what the platform already produced, and reasoned that nine task bindings at two days each, with some slack, comes to about ten working days. That reasoning is sound as far as it goes, and the bindings did in fact take eleven days.

The migration finished in the second week of the third month. The gap between ten days and eleven weeks is not a story about an engineer who estimates badly. It is a story about which line items were in the estimate, and the answer is that it contained exactly the work that is visible in a code editor.

Where eleven weeks went#

Here is the accounting from the retrospective, in engineer-days, with the original estimate beside it.

Two of the nine rows were estimated. Those two came in close. The other seven were absent from the estimate, and they are 78% of the work, which is the pattern rather than an anomaly of this particular team.

Inventory is first because you cannot revalidate what you cannot find#

Six days went into finding prompts. The task seam had accounted for the nine tasks in the main service, and the eval sets covered those nine. What it did not cover was a prompt embedded in a Retool query that the finance team used to categorise expense receipts, a prompt inside a nightly cron job that generated release notes, two prompts in a Jupyter notebook that the data team ran monthly and considered production, and a prompt in a Zapier automation owned by marketing. All five made model calls with the company's API key, none went through the platform, and three of them broke on the day the old key was rotated.

The technique that found them was billing rather than code search. Every API key in the provider console was mapped to an owner, keys with no owner were disabled for an hour, and the people who complained were the inventory. That is a blunt method and it found two dependencies that grep did not, because grep only searches repositories you know about.

Dual-running is where the schedule actually goes#

Once a candidate binding passes its eval set, the remaining question is whether it behaves the same on live traffic, which is not a question an offline case set can answer, since the case set was assembled from what someone thought to collect. The mechanism is to serve from the incumbent and run the candidate on the same input in parallel, discarding its output and recording the comparison.

# migration/shadow.py
async def serve(task: Task, payload: In) -> Out:
    primary = await task.run_with(BINDING[task.name].serving, payload)
    if random.random() < SHADOW_RATE[task.name]:          # 0.05 for high-volume tasks
        asyncio.create_task(_shadow(task, payload, primary))
    return primary.value

async def _shadow(task: Task, payload: In, primary: Result[Out]) -> None:
    try:
        candidate = await task.run_with(BINDING[task.name].candidate, payload)
    except Exception as exc:
        return _record(task, payload, primary, error=exc)
    _record(task, payload, primary, candidate,
            agree=task.equivalence(primary.value, candidate.value),
            d_latency_ms=candidate.latency_ms - primary.latency_ms,
            d_cost_usd=candidate.cost_usd - primary.cost_usd)

task.equivalence is the part that carries the design. For classify-ticket it is equality of the label, and the disagreement rate came out at 3.8% over 41,000 shadowed tickets. For extract-invoice it is field-by-field comparison with a one-cent tolerance on totals, giving 1.2% disagreement. For draft-reply there is no equivalence function that means anything, because two different well-written replies to the same customer are both correct, and the comparison has to be a human reading a pair of outputs with the provenance hidden.

That is where the fourteen days went. Of the 41,000 shadowed tickets, 1,558 disagreed; a stratified sample of 400 — drawn so that each category of ticket is represented in proportion rather than by luck of the draw — went to two support leads for blind adjudication, meaning they judged pairs of outputs without being told which model produced which, at roughly four minutes each, which is 27 hours of their time spread over three weeks of their availability. The result was that the candidate was right in 214 cases, the incumbent in 122, and 64 were judged as both acceptable. That outcome — candidate better on net, with a tail of cases where it is worse — is the normal outcome, and it is the input to a decision rather than a decision.

The shape matters more than the durations. The code work finishes in the first two weeks and then the schedule is entirely evidence, data and agreement, none of which can be parallelised by adding engineers, because the constraints are adjudicator availability, embedding throughput and a decision meeting that happens weekly.

Stored data is a migration of its own#

Seventeen days went to data, and the reason is that model outputs had been persisted and were being compared to each other.

The search index held 4.2 million embedding vectors produced by the incumbent's embedding model, an embedding being the list of numbers a model assigns to a chunk of text so that chunks with similar meaning land near each other, which is what makes a search over meaning rather than keywords possible. Vectors from two different embedding models are not comparable — cosine similarity between them is a number with no meaning — so the index could not be migrated incrementally by re-embedding documents as they were touched. It had to be rebuilt into a second index and cut over atomically. Re-embedding 4.2M chunks averaging 380 tokens is about 1.6 billion tokens, which at $0.02 per million is $32 of API spend and about nine days of wall-clock at the account's rate limit, plus index build and a period where both indexes are live and storage doubles.

Alongside that sat 380,000 cached task outputs: summaries shown in the UI, extracted invoice fields written to the accounting ledger, and classification labels used by routing rules. Those did not need to be identical to new outputs, but they did need to be consistent, because a support agent looking at two tickets classified six weeks apart should not see two different taxonomies. Reclassifying all 380,000 was rejected on cost grounds for the summaries and accepted for the labels, which is the kind of decision that only appears once someone asks the question.

Holding the rollback open costs five days and buys the schedule#

The five-day line item for rollback is easy to cut from a plan and is what makes the rest of the plan safe to execute. Keeping it open meant three things concretely. Both bindings stayed registered and both eval suites ran on every commit, so a prompt change made during the migration could not silently break the incumbent that was still serving traffic. The old provider's key stayed provisioned with enough quota to absorb full traffic, which required leaving a committed-spend tier in place for six weeks past the point where finance wanted to cancel it. And the flag that selects a binding was per task and changeable at runtime, so a rollback was a configuration write rather than a deploy.

That last property was used twice. classify-ticket went to full traffic on a Tuesday and was rolled back on the Wednesday when the routing team noticed that a downstream rule keyed on the exact string technical was receiving technical_support for about one request in six hundred — a case the eval set had not contained because the case set was built from labels rather than from raw model output. The rollback took ninety seconds, the fix took two hours, and the task went out again on the Friday. Without a live incumbent, the same discovery is an incident with a two-hour mean time to repair instead of a ninety-second one.

The trap in rollback planning is work that is not reversible. Once the new binding has written 40,000 extracted invoices into the accounting ledger, rolling the code back does not roll those records back, and if the two models disagree about date normalisation you now have a ledger with two conventions in it. The rule that came out of this migration was that any task writing to durable storage records the binding identifier alongside the row, which costs one column and makes the disagreement queryable afterwards instead of archaeological.

Estimating the next one before the work starts#

The table at the top of this chapter is a template as much as a record, and producing a defensible number for a migration means answering its rows in advance rather than reading the API reference. The questions that generate the estimate are: how many tasks, and do they all have eval sets; what fraction of traffic runs through code paths nobody has inventoried, which the billing sweep answers in a day; which tasks have an equivalence function and which need human adjudication, since the second group sets the calendar through other people's availability; what stored data was produced by the outgoing model and is compared against new output; and which decisions will need a person outside engineering to make.

An estimate that answers those five questions will still be wrong, and it will be wrong by a factor closer to 1.3 than to 4. The two-week figure was not a bad estimate of the work it described; it was an accurate estimate of one row.

The part nobody plans for#

The eight days at the bottom of the table were spent deciding what "good enough" means on tasks where the two models are different in ways that are not ordered.

draft-reply was the case. The candidate produced replies averaging 96 words against the incumbent's 148. Blind adjudication preferred the candidate 54% to 31%. CSAT — the customer satisfaction score, here the one-to-five rating customers leave after a support interaction — moved from 4.31 to 4.34 on a two-week holdout, which is inside the noise for that sample size. Two support leads argued that the shorter replies read as curt for billing disputes specifically, and produced eleven examples where they would have escalated. There is no threshold in any eval suite that adjudicates this, because the disagreement is about what the product should sound like, and that question had never been written down in a form precise enough to test against.

What resolved it was not more measurement. It was a decision, made by the head of support, that billing disputes would keep a length floor written into the task's prompt and everything else would move, plus an added eval gate asserting a minimum word count on the billing slice. Getting to that decision took four meetings across three weeks, and the reason it is worth pricing in an estimate is that it is not unusual — any migration of a generative task will surface at least one of these, and the cost is calendar time in other people's diaries rather than engineering effort.

What this estimate is not#

It is not a general multiplier to apply to any model change. A team with no stored embeddings, no persisted model outputs and only classification tasks with crisp equivalence functions would have skipped the data section and most of the adjudication, and finished in three weeks. The line items are the durable part; their weights depend on how much of your system has model output baked into it.

It is also not the cost of building the eval sets, which chapter 2 already charged to the seam. This migration inherited nine suites. A team starting cold adds that work on top, and it is usually the largest single item on the list.

And it is not the same exercise as a version change within one model family, where the API is identical, the SDK does not change, and the diff is one string in a config file. That case gets no schedule at all, because there is nothing visible to schedule, and the next chapter is about how much of the eleven weeks above still applies when the only thing that changed is the last segment of a model identifier.