Batches are assembled to a token budget, not an item count

The plan on the whiteboard was three bullets. Pack 2,048 chunks into each request, which keeps the job comfortably under the provider's 3,000 requests-per-minute cap. Run 32 workers. Retry 429s with exponential backoff. Six million chunks, a six-hour maintenance window on a Saturday, and a design that fits on an index card.

The first request failed before any of the rest of it mattered:

[04:11:02] POST /v1/embeddings  items=2048  tokens=819200
[04:11:02] 400 {"error":{"code":"max_tokens_per_request_exceeded",
                          "message":"Requested 819200 tokens, limit 300000"}}

After that was patched to 512 items and the job actually started moving, the second problem arrived around the fiftieth minute:

[05:02:11] workers=32 inflight=32  accepted=118  rejected=214
[05:02:11] goodput=3.06M tok/min  mean retry-after=8.4s
[05:02:11] p95 request latency=11.9s (9.1s of it spent in backoff)

Three million tokens a minute of goodput — the rate at which the provider was accepting work, as distinct from the rate at which the job was offering it — against a five million token quota, with two thirds of the requests being rejected before they were read. The job would not have finished in six hours at that rate, and it would not have finished in six hours at the theoretical maximum rate either, which is the thing worth establishing before designing anything at all.

Work out which cap binds, in two lines#

Six million chunks at a mean of 400 tokens is 2.4 billion tokens. Against a 5 million tokens-per-minute quota, that is 480 minutes, or 8.0 hours, and that is a floor rather than an estimate — it assumes perfect packing, zero rejected requests, no production traffic sharing the quota, and no time spent doing anything except moving tokens.

The request cap is nowhere near binding, and the arithmetic showing that takes one more line. Sustaining 5 million tokens per minute through 3,000 requests per minute requires only 1,667 tokens per request, which is between four and five chunks. Any batch size above five chunks makes the token cap the constraint; at a realistic 250 chunks per request, the request cap would permit 300 million tokens per minute, which is sixty times the token quota. Designing against the requests-per-minute number, as the whiteboard plan did, is optimising the constraint that has 60x of headroom.

That gives the honest answer about the six-hour window before a line of code is written: it is not achievable at this quota, and no batching strategy or concurrency scheme will make 2.4 billion tokens pass through a 5 million tokens-per-minute gate in 360 minutes. The options are to raise the quota, to run against a second project with its own quota, to move part of the corpus onto self-hosted inference, or to change the window. The provider's async batch endpoint — which topic 2.13 covered as a general pattern, and which typically costs half the synchronous price — is ruled out by the deadline rather than by the economics, since its turnaround is measured in hours to a day. Saying this on the Thursday costs a conversation. Discovering it at 07:00 on the Saturday costs the window.

Packing to a token budget#

Once the token cap is known to be the binding one, the batch is assembled by summing actual token counts rather than by counting items. The per-request token limit is a hard edge and a request that crosses it is rejected whole, so the entire batch's round trip is wasted rather than the excess being trimmed.

MAX_TOKENS_PER_REQUEST = 300_000
MAX_ITEMS_PER_REQUEST = 2_048
TARGET = int(MAX_TOKENS_PER_REQUEST * 0.80)   # headroom for tokeniser disagreement
MODEL_CONTEXT = 8_191


def pack(chunks):
    """chunks: iterable of (chunk_id, text, token_count), pre-truncated."""
    batch, tokens = [], 0
    for chunk_id, text, n in chunks:
        if n > MODEL_CONTEXT:
            raise ValueError(f"{chunk_id}: {n} tokens exceeds model context")
        if tokens + n > TARGET or len(batch) == MAX_ITEMS_PER_REQUEST:
            yield batch, tokens
            batch, tokens = [], 0
        batch.append((chunk_id, text))
        tokens += n
    if batch:
        yield batch, tokens

The 80% target is not timidity. Your tokeniser and the provider's can disagree by a few percent on text containing unusual scripts, long URLs or base64 fragments, and a batch computed at 299,000 tokens locally that the provider counts as 304,000 is a whole rejected request. The item cap stays in the loop as a secondary constraint because it is also a hard edge, and on a corpus of very short chunks it becomes the binding one — 2,048 chunks of 40 tokens is 82,000 tokens, well under the token budget, so a purely token-driven packer would try to put 7,500 items in a request that the API will not accept.

The MODEL_CONTEXT check matters more than it looks. Embedding models truncate or reject inputs above their context length, and a single 12,000-token chunk in a batch of 250 takes the entire batch down with it. Chunks longer than the model context should be caught in the chunking stage, but corpora contain surprises — a document that is one 40,000-token XML blob with no structural breaks for the chunker to find — and a batch job that dies at hour three on a single pathological input is a bad afternoon.

Packing by tokens also changes the shape of the work. With chunks averaging 400 tokens and a 240,000-token target, batches average 600 items, and 6 million chunks become 10,000 requests. At 8 hours that is 21 requests per minute, which puts the requests-per-minute cap even further out of the picture and makes clear that the entire job is 10,000 round trips rather than 6 million.

A batch is a unit of failure as well as a unit of work#

Larger batches are more efficient and fail more expensively, and the trade is worth making explicitly. Most embedding APIs validate the whole request before processing any of it, so a single malformed input — an empty string, a lone surrogate pair, a chunk that exceeds the context length — rejects all 600 items with one error message that names none of them. Retrying the batch unchanged reproduces the failure forever, which is how a job ends up in a retry loop that burns quota without moving a single chunk.

The handling is a bisect. On a 400-class error that is not a rate limit, split the batch in half, submit both halves, and recurse on whichever fails; a poisoned item in a 600-item batch is isolated in about ten requests, quarantined to a dead-letter table — the side table where items that cannot be processed are parked with their error for someone to look at later — and the other 599 proceed. Without that, the usual response is to shrink the global batch size until the job gets past the bad input, which multiplies request count for the whole run to work around one document.

Measuring progress correctly matters for the same reason. Requests per second is the wrong meter, because a job whose batches are shrinking looks faster while doing less; accepted tokens per minute is the meter that maps onto both the quota and the completion date. Track it alongside three counters that explain it — rejected requests, tokens in flight, and current concurrency — and a collapse becomes readable at a glance rather than something inferred from a flat progress bar.

Where there is no hard deadline, the arithmetic changes enough to be worth re-running. The asynchronous batch endpoint takes a file of requests, returns results within a stated turnaround, and prices at about half the synchronous rate, and crucially it does not draw on the same tokens-per-minute quota, so it does not compete with production ingest at all. For 2.4 billion tokens that is $120 instead of $240 at $0.10 per million, with no concurrency controller to write and no shared limiter to negotiate. The reason it is ruled out here is the six-hour window and nothing else, which makes "does this actually need to finish tonight?" the cheapest question on the list.

Controlling concurrency rather than retrying into it#

Thirty-two workers against a token ceiling do not move more tokens than eight workers do, because the ceiling is enforced on tokens and not on connections. What the extra workers add is rejected requests, each of which costs a round trip, and uncoordinated exponential backoff synchronises them into waves, so the quota sits underused between waves and oversubscribed during them. The measured goodput of 3.06 million tokens per minute against a 5 million quota is the signature of that pattern.

The correction has two parts. A client-side token bucket — a counter that accrues allowance at a fixed rate, is decremented by each request's token count, and makes a caller wait when it has run ahead of its share — refilled at your allotted portion of the quota, means the ceiling is respected without ever needing a 429 to discover it. An additive-increase, multiplicative-decrease controller on concurrency handles everything the token bucket cannot know about — other consumers of the same quota, provider-side capacity shifts, per-model limits that differ from the account limit.

import time

class AdaptiveConcurrency:
    def __init__(self, initial=4, floor=1, ceiling=64, cooldown_s=60):
        self.limit = initial
        self.floor, self.ceiling = floor, ceiling
        self.cooldown_s = cooldown_s
        self.clean = 0
        self.last_throttle = 0.0

    def on_success(self):
        self.clean += 1
        if self.clean >= 30 and time.monotonic() - self.last_throttle > self.cooldown_s:
            self.limit = min(self.ceiling, self.limit + 1)
            self.clean = 0

    def on_throttle(self, retry_after: float):
        self.clean = 0
        if time.monotonic() - self.last_throttle > self.cooldown_s:
            self.limit = max(self.floor, self.limit // 2)
        self.last_throttle = time.monotonic()
        return max(retry_after, 1.0)

The cooldown is what stops a burst of thirty simultaneous 429s from halving the limit thirty times and taking it to one. Only the first throttle in a window is allowed to cut; the rest are absorbed. Rebuilt this way on the same corpus and quota, the job settled at a concurrency of 9 and 4.6 million tokens per minute sustained, which is 92% of quota and a 50% improvement in goodput from removing workers.

What production ingest is owed#

The quota is shared with the live ingest path, which embeds new and edited documents as they arrive and has a latency expectation attached to it. A backfill that consumes the whole quota does not fail; it makes the product look broken, because newly published documents stop appearing in search for hours.

The reserved-floor arrangement is the one that survives contact with reality most often: production gets a guaranteed 1.2 million tokens per minute whether it uses them or not, the backfill draws from the remaining 3.8 million, and the backfill's schedule is computed against 3.8 rather than 5, which takes the floor from 8.0 hours to 10.5. Publishing that number as the plan is better than publishing 8.0 and explaining the difference afterwards.

The self-hosted row deserves a second look on large backfills specifically. A single mid-range GPU running an open embedding model handles on the order of 300 sequences per second at these lengths, which is 7.2 million tokens per minute — above the API quota — and four of them clear 2.4 billion tokens in about 1.4 hours for a few dollars of instance time. The catch is total: whatever model embeds the corpus must also embed every query for as long as that index lives, so this is a decision about the retrieval stack rather than a scheduling trick.

The part concurrency control does not touch#

Everything here is about the rate at which work is admitted, and none of it says anything about what happens when the job stops. A well-behaved controller running at 92% of quota for ten hours is still a job that can die at hour seven with 4.1 million chunks embedded and no record of which ones. Whether that is a twenty-minute inconvenience or a second ten-hour run is decided by a property of the identifiers the job writes, and it has to be decided before the job starts.