Embedding pipeline economics

Read these first

• Index maintenance
• Batch and async job endpoints

What you will know

• Estimate the cost of embedding a corpus before running it

  Use representative token counts, provider pricing, retry assumptions, and storage volume to produce a budget with an uncertainty range.

• Re-embed incrementally rather than wholesale where possible

  A changed parser, chunking rule, or embedding configuration can invalidate vectors even when source timestamps appear unchanged.

• Batch efficiently against rate limits

  Respect provider item and token caps while handling throttling, partial failures, idempotency, and adaptive concurrency.

• Plan a full reindex as a costed project

  Include staging capacity, temporary duplicate indexes, validation, cutover, rollback, and the period when old and new vectors coexist.

• Calculate and compare the cost of a model change at your corpus size

  Account for re-embedding, storage, evaluation, dual-serving, and downstream query-quality testing rather than counting API calls alone.

Chapters

1. 1

   The invalidation key is a fingerprint of the text you embedded

   You change the HTML-to-text extractor to strip navigation chrome. The CMS documents are untouched, so their updated_at timestamps have not moved, and the incremental job selects on updated_at and reports zero work while the index still encodes the chrome. Move invalidation onto a fingerprint of the text actually sent to the model, combined with the chunking and model configuration, so a silent pipeline change cannot leave stale vectors behind unnoticed.

   Read chapter
2. 2

   Change the chunker and there is no affected subset to compute

   In one week a content team edits 40,000 of your 8M documents and you move chunk size from 512 tokens to 256 with 20% overlap. The first is an incremental re-embed; the second is a full reindex, because chunk size is a property of the pipeline rather than of any document. Every document is re-chunked, every chunk id changes, and overlap rewrites the text of every chunk touching a boundary. Telling the two kinds apart is what makes a quote credible.

   Read chapter
3. 3

   A defensible embedding quote is a range built from a stratified sample

   Someone asks what embedding a 40M-document corpus will cost. The number worth defending comes from a sample stratified by document length and content type, run through the production chunker, priced at list rate, with a partial-failure allowance and a storage figure, and given as a range. A quote of $18,000 that lands at $47,000 is almost never retries — rate-limited requests are not billed — but chunking overhead, which is a property of the pipeline no sample of documents can reveal, and a length distribution the sample never contained.

   Read chapter
4. 4

   The stored chunk text usually outweighs the vectors themselves

   Dimensions times bytes per component gives the raw vector; the ANN graph or quantisation codebook layers on top of it; the chunk text and metadata kept beside each vector routinely exceed both; and every replica multiplies the total. Work that arithmetic for your own corpus and the two figures this topic leans on stop being assertions you have to take on trust: why 1536 dimensions dropping to 1024 removes less of the bill than it looks like it should, and why carrying two indexes through a cutover runs near 1.7 times today's footprint rather than 0.67.

   Read chapter
5. 5

   Batches are assembled to a token budget, not an item count

   Six million chunks averaging 400 tokens, a 5M tokens-per-minute cap and a 3,000 requests-per-minute cap: establish which one binds before designing anything. Packing 2,048 items per request blows the per-request token limit and is rejected whole, and thirty-two workers pushed against a token ceiling sit in backoff while goodput falls. Size batches by tokens, halve concurrency on a 429, creep back on sustained success, and decide what production ingest sharing the quota is owed.

   Read chapter
6. 6

   Deterministic chunk ids turn a resume into a set difference

   A twelve-hour backfill dies at hour nine, having written vectors as it went and kept no checkpoint file. Whether you resume or restart rests on one property: whether the vector store can tell you which chunk ids already carry a current vector. Derive the id from the document and the chunking configuration and the remaining work is a set difference; derive it from a counter or a random identifier and the answer is another twelve hours.

   Read chapter
7. 7

   The migration bill is the window where both indexes are live

   A model scoring four points higher and costing 30% less per token reads as a saving until the transition is priced. Re-embedding 300M chunks is the smallest line: the total is dominated by carrying two indexes through validation and cutover, where storage and ANN memory roughly double, plus the dual-serving window, rollback capacity, and the query-quality evaluation proving those four points survive on your corpus. A steady-state saving arrives after the expensive part rather than instead of it.

   Read chapter
8. 8

   Four benchmark points are an assumption until your own queries test them

   Everything in the previous chapter is spent on the premise that a model scoring four points higher in public is better on your corpus, and month one of a nine-month deprecation window is when you find out rather than when you provision. Stand up a query set with relevance judgements over a corpus slice large enough to separate the two models, fix the metric before you look at anything, and decide in advance which result aborts the migration rather than merely delaying it. Capacity commitments follow that answer; they do not precede it.

   Read chapter