The batch job round trip

A batch endpoint is an asynchronous interface to the same models an application would otherwise call one request at a time. You hand the provider a whole collection of requests in a single call — 380,000 product descriptions, each carrying its own prompt — and what comes back immediately is a job identifier rather than an answer, together with an undertaking that the work will be done within a stated window, usually twenty-four hours. The job then runs on the provider's infrastructure, whether or not anything of yours is still alive to watch it. Anthropic's version is called the Message Batches API and OpenAI's the Batch API, API standing for application programming interface; both sit on the same HTTP surface as the synchronous endpoints, meaning the ordinary call where the client holds the connection open and waits a few seconds for the model to reply.

The arrangement exists because inference capacity is uneven across a day and providers would rather fill the troughs than leave them empty, and a customer with a million requests and no opinion about which hour they run in fills troughs nicely. That sentence is a conclusion, and the steps under it are what make the rest of the arrangement predictable rather than arbitrary. Serving capacity is bought and provisioned ahead of the demand it will meet, and an accelerator-hour nobody uses cannot be stored and sold tomorrow — it is spent whether or not any work ran on it. So work carrying no opinion about which hour it runs in can be moved into the hours nobody else wants, and serving it there costs the provider less at the margin than serving work that has to run the moment it arrives.

The trade is priced accordingly: you give up any say in timing, and in return the tokens are billed at half the standard rate and throughput management stops being your problem, since you are no longer the one deciding when each request goes out. The half is a share of the saving described above rather than a discount on a lesser model, and the thing handed over in exchange is exactly the freedom that produced the saving. What is not handed over is the model. A request inside a batch is served by the same weights at the same tokens per second as the same request sent synchronously, so nothing about the computation differs; what you wait for is scheduling, and scheduling happens before any computation starts.

This chapter is about the mechanics of one complete round trip, and about the single property that makes those mechanics different from an ordinary HTTP call with a long timeout: the job outlives the process. A synchronous request that dies with its process costs one retry. A batch job that dies with its process costs every request in it, because the work continues, the bill continues, and the only handle you had to reach it is gone. The running example throughout is an overnight product enrichment job — 380,000 descriptions in, structured attributes out — which is the shape of workload these endpoints exist for, alongside backfills that re-label archived data and evaluation runs that score thousands of saved transcripts.

Outliving the process is a mechanical consequence rather than a design flourish, and seeing why removes most of the surprise from everything after it. The create call is a write into the provider's own job store: when it returns, a record exists on their side describing work they have undertaken to do. Nothing in HTTP ties a server-side record to the lifetime of the connection that created it, and nothing in the batch protocol asks a client to hold anything open. So killing the client, draining the node or resetting the connection changes nothing about the record, and the identifier that came back in the response is the only reference to it that will ever exist outside the provider.

That is worth stating against the instinct it contradicts, because the instinct is a sound one everywhere else. In every other kind of work an engineer owns — a thread, a Celery task, a Kubernetes job, a cron invocation — the process is the work, so stopping the process is a real way of stopping the work. Here the process is a client of the work rather than the work itself, and from inside a polling loop the two situations look identical.

The round trip has four moving parts, and only one of them is the call#

Submitting a batch is not one API call with a long timeout. It is a submit, then an independent job that outlives your process, then a retrieval of a results artifact, then a reconciliation against what you sent. Each of those is a separate failure domain, and the code has to be written as if the process will die between any two of them.

Start with the request file, because its shape constrains everything downstream. Anthropic's Message Batches API takes a list of request objects in the POST body, each one carrying a custom_id and a params object that is exactly the body you would have sent to /v1/messages:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY, or an `ant auth login` profile

PROMPT = (
    "Extract product attributes from the description below and return JSON "
    "with keys material, colour, dimensions_cm. Use null for anything absent.\n\n"
)

requests = [
    {
        "custom_id": f"product-{row.product_id}",
        "params": {
            "model": "claude-opus-5",
            "max_tokens": 512,
            "messages": [{"role": "user", "content": PROMPT + row.description}],
        },
    }
    for row in rows
]

batch = client.messages.batches.create(requests=requests)
print(batch.id, batch.processing_status)  # msgbatch_01HxQ2m4kZ8vT3nRbY7cJp9L in_progress

OpenAI's Batch API splits the same information across two calls. You write a JSONL file — JSON Lines, one self-contained JSON object per line — where every line is a complete HTTP request — method, url, body — with a custom_id at the top level, upload it with purpose batch, then create the job against the uploaded file id:

{"custom_id": "product-84120", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1-mini", "max_tokens": 512, "messages": [{"role": "user", "content": "Extract product attributes..."}]}}
{"custom_id": "product-84121", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1-mini", "max_tokens": 512, "messages": [{"role": "user", "content": "Extract product attributes..."}]}}

from openai import OpenAI

oai = OpenAI()
uploaded = oai.files.create(file=open("nightly-2026-03-14.jsonl", "rb"), purpose="batch")
job = oai.batches.create(
    input_file_id=uploaded.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

Both shapes exist to serve the same constraint: the provider is going to run these requests in whatever order its scheduler prefers, across however many workers it has free, and hand you back a file whose line order tells you nothing. The custom_id is the only thing tying a result to the row it came from. It is not a convenience field. If you set it to a positional index and your input query is later ordered differently, or if you leave two rows sharing an id, you have destroyed the join, and the only way to find out is a data-quality problem discovered three weeks later.

Why line order tells you nothing deserves one more step, because a reader who suspects the provider could have preserved it will treat custom_id as belt and braces. The requests are spread across however many workers are free when the job is scheduled, each request takes a different amount of time to answer, and the results file is written as answers arrive. Line order is therefore completion order, and completion order carries no information about submission order. The only way to restore the original sequence would be to hold every answer until the last one lands, which is precisely the buffering an asynchronous endpoint exists to avoid.

A custom_id set to a positional index therefore fails in the way that is hardest to catch. Every line still finds a row, every row still receives attributes, and nothing in the load reports an error — the join is well-formed, and it is joining the wrong pairs. A join that breaks loudly costs an evening. A join that stays valid and wrong is found by somebody looking at a product page a month later and wondering why an oak table is described as waterproof.

Give the custom_id a shape you can parse and that carries enough context to be useful in a log line: product-84120, or run-2026-03-14/product-84120 if you submit the same entity across multiple runs and want the results file to be self-describing. Both providers cap it at a few hundred characters, so there is room. Resist the temptation to pack the whole row into it — the id is a join key, not a payload, and anything you put in it is duplicated across every request in the file, which counts against the 256 MB body limit. That limit applies to the submission as a whole rather than to any one request in it, so a field that looks negligible per row is multiplied by the row count before it meets the ceiling.

Two habits are worth building here. The first is asserting uniqueness before you submit rather than discovering a collision in the results: assert len({r["custom_id"] for r in requests}) == len(requests) costs microseconds and catches the class of bug where a de-duplication step upstream silently stopped working. The second is writing the submitted ids somewhere queryable at submit time, not just the count. A run that recorded request_count = 380000 can tell you how many results are missing; a run that recorded the ids can tell you which, and that difference is what the reconciliation work in later chapters is built on.

Authentication is the least interesting part and the one most likely to be quietly wrong. Batch endpoints use the same credential as the synchronous endpoints — ANTHROPIC_API_KEY, or the OAuth profile that ant auth login writes to ~/.config/anthropic/ and the SDKs pick up automatically. What differs is the limit that governs you. Batch submissions draw on a separate enqueued-token allowance — a ceiling on how many tokens you may have sitting in queued jobs at once — rather than the per-minute request and token limits a worker pool contends with, which is why a batch submission can accept 100,000 requests in one call without any client-side pacing. It is also why a batch submission can fail on a limit your synchronous path has never touched, and why a 429 on submit means something different from a 429 in a request loop.

The difference between the two limits is what makes those two rejections different pieces of information. A per-minute request or token limit governs work the provider must serve now, and the client's remedy is to slow down: send fewer requests this minute and the same work fits. An enqueued-token allowance governs how much of your work may be waiting, and a submission is one indivisible call. There is no slower way to make it, because the whole collection either fits under the ceiling or it does not. A 429 in a request loop is therefore an instruction to pace, and a 429 on submit is a statement that the shape of the submission is wrong — split it, drain something already queued, or wait for a running job to finish.

Persist the id before you do anything else with it#

The instant create returns, you hold the only durable handle to hours of work that is already accruing charges. Write it down before you log it, before you print it, and certainly before you enter a polling loop:

with db.transaction():
    db.execute(
        """
        insert into batch_runs (run_id, provider, provider_job_id, submitted_at,
                                request_count, status)
        values (%s, 'anthropic', %s, now(), %s, 'submitted')
        """,
        (run_id, batch.id, len(requests)),
    )

That one row changes the recovery story completely. A process that dies after the insert leaves behind a job that any later process can find, poll and drain. A process that dies before it has to be assumed to have submitted, because create may well have succeeded on the provider's side and failed on the way back to you — which is an argument for generating run_id yourself, writing a submitting row first, and updating it with the provider's id once you have one. The cost of the extra write is a millisecond. The cost of an orphaned 380,000-request job is the whole run.

Polling itself is unremarkable once the id is durable. The loop reads the id from the database rather than from memory, and the loop being interrupted is a normal event rather than a disaster:

def drain(run_id: str) -> None:
    row = db.fetchone("select provider_job_id from batch_runs where run_id = %s", (run_id,))
    batch = client.messages.batches.retrieve(row.provider_job_id)

    if batch.processing_status != "ended":
        return  # scheduler will call us again; nothing was lost by stopping here

    raw_key = f"batch-output/{run_id}.jsonl"
    with s3_writer(raw_key) as out:
        for line in client.messages.batches.results(row.provider_job_id):
            out.write(line.model_dump_json() + "\n")

    db.execute(
        "update batch_runs set status = 'output_stored', raw_output_key = %s where run_id = %s",
        (raw_key, run_id),
    )

The important move in that function is the order of operations at the end. The raw results go to object storage, verbatim, before a single line is parsed. Parsing is where your schema assumptions live, and schema assumptions are the thing most likely to be wrong on the night a model produces something you did not anticipate. If the parse throws, you want to fix the parser and re-run it against a local file, not re-run 380,000 requests.

The order matters because of what a parse failure would otherwise take with it. The bytes the provider streamed are the artifact the money bought, and until they are written down they exist only inside a function that is about to raise. Run the parse first and an exception in twenty lines of your own code and the loss of the whole night's output become the same event. Store first and those two failures come apart: the parser may throw as often as it likes against a file that is not going anywhere, and each attempt costs a rerun of a script rather than a rerun of the job.

Anthropic keeps batch results available for 29 days and OpenAI keeps the output file until you delete it, which sounds like it makes local storage unnecessary; it does not, because that window is a grace period for recovery, not an archive you control, and it disappears the moment you need to re-parse something from last quarter. The reading to guard against is that the provider is already holding the artifact on your behalf. It is tempting because it is true for as long as it lasts, and it is wrong because none of the terms are yours: the duration is set by somebody else, the only way to reach the file is an id you must not lose, and the expiry has no relationship to when a column will next need re-deriving. The whole round trip, with every durable write marked, is this:

Read that diagram as three restartable segments rather than one flow. Between any two arrows the worker can vanish and a later invocation, holding nothing but run_id, can pick up exactly where it stopped.

Once the loop is restartable, the natural next step is to stop running a loop at all. A worker that sleeps for hours is a worker holding a pod, a database connection and a deploy lock hostage for the duration, and it will be killed eventually by something. Schedule drain(run_id) every few minutes from whatever already schedules work in your stack — a cron entry, an Airflow sensor, a Lambda on a rate rule — and let each invocation exit in milliseconds when the status is still in_progress. The provider charges nothing for status checks, and a fifteen-minute polling interval on a job whose realistic completion spread is measured in hours costs you nothing in freshness. The shape of the code stops being "wait for a slow call" and becomes what it always was underneath: a small state machine advanced by an external scheduler.

Two providers, one shape, different field names#

The two APIs differ enough to be annoying and not enough to be interesting. Keeping the differences in one table beats scattering provider conditionals through the code that calls them.

The one row that catches people is the terminal signal. ended on Anthropic and completed on OpenAI are both reached whether the requests inside succeeded or not, and OpenAI additionally reaches completed when some rows failed and were written to error_file_id. Neither value means "your data is good". They mean the provider has stopped working on the job and there is a file to fetch.

A batch endpoint is not a job queue#

You cannot re-prioritise a submitted job, you cannot inspect an individual request in flight, and you cannot pause it. The only verbs are submit, check, fetch and cancel. If your mental model is Celery or SQS, adjust it: this is closer to sending a tape to an off-site processor than to enqueueing work you still control. In particular, there is no server-side notion of your run. The provider knows about a job id; the mapping from that id to the 380,000 rows you meant to enrich exists only in your database, which is why the submit-time write is load-bearing rather than a logging nicety.

The gap that write leaves open is the one case where create succeeds on the provider's side and the response never reaches you — a connection reset, a proxy timeout, a pod killed mid-call. The submitting row survives that, but it holds no provider id, so recovery means listing recent jobs on the provider and matching one against the row by creation time and request count. Make the submit step idempotent where the provider allows it, and alert on any submitting row older than a few minutes.

That gap can be narrowed and cannot be closed, and the reason is worth being precise about rather than filing under bad luck. Two writes are involved — one in the provider's job store, one in your database — and they belong to separate systems with no shared transaction between them. There is no instant at which both are known to have happened. The provider's response is the only evidence that the first write occurred, and the response is exactly what a reset connection destroys. Everything in this chapter is therefore an arrangement for making that ambiguous window small and recoverable rather than for removing it: the submitting row bounds what a death inside the window can cost, the client-generated run_id gives the surviving row something to be matched against, and the alert bounds how long the ambiguity can sit there unnoticed.

None of the machinery in this chapter is durability for your own results. Persisting the job id protects the handle; storing the raw output protects the artifact. Everything after that — the parse, the schema check, the write into the products table — is ordinary data-loading code with ordinary data-loading failure modes, and the provider offers you nothing there.

A batch job is not a slow function call, and every mechanism in this chapter follows from taking that literally. The identifier is the whole of your claim on work that is running and billing on infrastructure you cannot see, so it goes to durable storage in the same breath as the submission that produced it, and the polling loop becomes a small scheduled step that can be killed at any moment without losing anything. The raw output file gets the same treatment: it is the artifact you paid for, and parsing is a decision you may want to make again.

None of which says anything about whether batch was the right choice. The enrichment job in the scenario recovered and wrote its 380,000 rows — a day after the window the data was needed for. That leaves the question the next chapter has to answer: what a batch endpoint actually promises about when results come back, and what you have measured rather than assumed.