Partial failure within a batch

A batch job of 500,000 requests can report success while 499,999 of the requests inside it failed. That is not a defect in the API: a single job-level state cannot summarise half a million independent outcomes without lying about most of them, so both major providers decline to try. Anthropic reaches processing_status: "ended" and OpenAI reaches status: "completed" when the work has stopped and a results file exists, and neither value is downstream of what is in that file. The check most data-loading code performs first is therefore the one check that cannot detect the failure it should care about.

This chapter is about what the file actually contains and how to read it. It separates the three kinds of damage a batch can carry, each with a different cause and a different correct response; it replaces "parse the results" with "assign every line a terminal state, then extract payloads only from the lines that earned one"; and it builds the reconciliation table that the rest of this topic depends on — a row per custom_id, the per-request identifier attached at submit time, written from the ids you sent rather than the ids that came back. The worked example is an overnight product-enrichment job whose output is written into a products table, and a review-scoring run of two million items that came back on time and 2,847 rows short.

The status field answers a question about the job, not about your data#

The argument that one status field cannot summarise half a million independent outcomes is true, and it is not the whole reason. Even a provider willing to compute a per-item verdict could not compute the one that matters, because the two questions are answered from different information. A job's status reports the condition of the provider's own work queue — whether there is anything left to schedule for this job — which is a fact the provider owns and can settle exactly. Whether a response body satisfies your schema is a fact about a document the provider has never seen and holds no definition of: the schema lives in your code, may have been written after the job was submitted, and can be tightened tomorrow without anyone being told. No amount of care on the provider's side could make the first answer the second, which is why the gap is there by construction rather than by oversight.

What the providers do give you is a count. Anthropic returns request_counts with processing, succeeded, errored, canceled and expired; OpenAI returns total, completed and failed and writes the failures to a separate error_file_id. Reading those counts costs one API call and is the cheapest possible smoke test, so read them. They are still not sufficient, because they cannot see the third kind of damage: a request the provider served successfully, billed you for, and whose body your schema rejects. As far as the provider is concerned that row succeeded.

Three kinds of damage, then, with different causes and different correct responses:

The middle row is the one people forget exists, because it has no analogue in synchronous work. Where the missing analogue would have to come from is worth spelling out. In a synchronous call, acceptance and execution are the same event: the provider either serves the request or refuses it at the door, and there is no interval in which it holds work it has taken on and has not yet decided to do. A batch separates those two events by hours. The moment they are separated a third outcome becomes available — work accepted, never started, handed back — and a request that never ran is not an error and not a success; it is an absence with a receipt.

The correct response follows from that and is the one most often got wrong. Routing expired items into the same triage as provider errors sends somebody looking for a fault in a request that was fine, and that search ends only when they give up on it.

The last row is the one that costs money twice, since you paid for the tokens and you will pay again for the retry. It is also the only one of the three that your own code makes more or less common, because it is defined by a schema you wrote rather than by anything the provider did or failed to do.

Reading the output as outcomes, not as data#

The parse step has one job before it has any other: assign every line to a terminal outcome and write that outcome down. Extracting the payload comes after, and only for the lines that earned it.

from pydantic import BaseModel, ValidationError

class Attributes(BaseModel):
    material: str | None
    colour: str | None
    dimensions_cm: list[float] | None

def classify(line) -> tuple[str, dict | None, str | None]:
    """Returns (terminal_state, payload, detail)."""
    result = line.result
    if result.type == "expired":
        return "expired", None, None
    if result.type == "errored":
        return "provider_error", None, result.error.type
    if result.type == "canceled":
        return "canceled", None, None

    text = result.message.content[0].text
    try:
        payload = Attributes.model_validate_json(text).model_dump()
    except ValidationError as exc:
        return "schema_failure", None, exc.errors()[0]["type"]
    return "succeeded", payload, None

Notice what classify does not do: it never raises, and it never returns a partially-populated payload. Every path out of it produces a state the calling code knows how to act on. A bare except around json.loads that returns None is not careless so much as it answers the wrong question — it asks "can I parse this line" when the question is "what happened to product 84120".

That substitution deserves a defence before it gets a verdict, because writing it is a reasonable act. A parse failure genuinely feels like a value problem: the text did not parse, so there is no value, and None is an honest return from a function whose job is turning text into an object. What makes it wrong is the destination rather than the reasoning. Further down the pipeline None also means "the model looked and found nothing to record", which is a legitimate outcome on some rows, so at the moment of the write a failed parse and an empty answer become the same row. Nothing downstream can separate them afterwards, because the distinction was destroyed by the write rather than lost by it. A terminal state keeps them apart by refusing to collapse two different histories into one value.

Anthropic streams results as objects, so the shapes above are attributes on a parsed model. OpenAI hands you a JSONL file — JSON Lines, one self-contained JSON object per line — where the successes and the failures are in two different files, and a successful line looks like this:

{"id": "batch_req_9tK2", "custom_id": "product-84120", "response": {"status_code": 200, "request_id": "e1c2", "body": {"choices": [{"message": {"content": "{\"material\":\"oak\",\"colour\":\"natural\",\"dimensions_cm\":[120,45,74]}"}}]}}, "error": null}

with the error key populated and response null on the failures. The two providers differ in packaging, but the classification is identical, which is worth exploiting: keep classify provider-specific and small, and keep everything after it provider-agnostic.

The reconciliation table is the unit of work#

Once every line has a state, the state belongs in a table keyed by the custom_id you sent, and the table has to be populated from the ids you submitted rather than from the ids that came back. That inversion is the whole trick. If the table is built by iterating the output file, a request whose line is missing entirely — which happens, rarely, when a results stream is truncated by a network fault mid-download — leaves no row and is invisible. If the table is built at submit time, a missing line shows up as a row still sitting in pending after the job ended, which is a state a reconciler can see.

The general form of that is worth carrying to other problems, because it is not about batches. Detecting a missing element requires possession of the complete expected set, since absence is not a property that any surviving record carries — nothing in the output file says "and one more should have been here". Iterating the output gives you exactly what arrived, so a line that never arrived has nothing to be iterated over, and its absence is not merely hard to see but unrepresentable. Writing the rows at submit time converts that absence into a value sitting in a column, and a value is something a query can return. In practice that is one row per submitted item:

create table batch_items (
    run_id          text        not null,
    custom_id       text        not null,
    input_hash      text        not null,
    state           text        not null default 'pending',
        -- pending | succeeded | provider_error | expired | schema_failure | canceled | abandoned
    attempts        smallint    not null default 1,
    last_detail     text,
    settled_at      timestamptz,
    primary key (run_id, custom_id)
);

create index on batch_items (run_id, state) where state <> 'succeeded';

The partial index matters more than it looks. On a 500,000-row nightly run the interesting set is the few thousand rows that are not succeeded, and every query the reconciler and the retry driver run is a filter on exactly that predicate. A partial index over the unsettled rows stays small — a few hundred kilobytes against a table with half a million entries — and keeps the "what is outstanding" query at a couple of milliseconds even when the run has been retried four times.

The size claim is the part worth checking, because it is where the technique earns its keep. An index costs storage in proportion to the number of rows it holds rather than to the number of rows in the table it points at, and the predicate state <> 'succeeded' admits only the unsettled few thousand out of the half million. That makes it two orders of magnitude smaller than an index over the whole table, small enough to stay resident in memory, and — unusually — something that shrinks as the run progresses, because every retry that succeeds removes a row from the predicate rather than adding one.

input_hash is there so that a retry can prove it is sending the same request it sent the first time. When an enrichment prompt changes between the original submission and the retry, the retried rows are enriched under different instructions from their neighbours, and nothing in the data records that. Hashing the rendered request body at submit time and comparing on retry turns a silent inconsistency into a loud one.

The failure it catches is a scheduling accident rather than a bug anyone would write deliberately. A retry is built by today's code from today's prompt template, and the original was built by whatever code and template were deployed on the night the job ran. Between those two moments there may have been a release. Both requests are well-formed, both come back as successes, and the only difference between them is which instructions produced the attributes — which is invisible in the output and permanent in the table. A hash comparison is the cheapest thing that can notice, because the hash is the only artifact of the first submission that survives to be compared against.

The loader then reads from this table rather than from the file, and the change in the write is small and decisive:

update products p
   set attributes = i.payload,
       enriched_at = now()
  from batch_item_payloads i
 where i.run_id = %(run_id)s
   and i.state = 'succeeded'
   and p.product_id = split_part(i.custom_id, '-', 2)::bigint;

Nothing in that statement can write a null over a good value, because rows that are not succeeded are not in the result set. The original loader's fault was not that it mishandled errors — it had no error-handling code to get wrong. It was that it wrote unconditionally, so an error and a legitimately empty attribute set were indistinguishable by the time the write happened. This version never has to tell them apart, because the filter removes one of the two before the write is reached.

Retrying 4,300 rows instead of 120,000#

A batch of 120,000 requests comes back with 3,400 per-item provider errors and 900 schema failures. The two figures that decide the retry are each one arithmetic step from those counts: 3,400 and 900 sum to the 4,300 outstanding items that still need work, and taking 4,300 away from 120,000 leaves the 115,700 settled ones that do not. The retry is a new batch containing exactly those 4,300 custom ids, submitted through the same path as the original — same submit-time insert, same durable job id, same raw output storage — with the reconciliation table updated rather than replaced.

outstanding = db.fetchall(
    """
    select custom_id, input_hash
      from batch_items
     where run_id = %s
       and state in ('provider_error', 'expired', 'schema_failure')
       and attempts < 3
    """,
    (run_id,),
)
retry = client.messages.batches.create(
    requests=[build_request(row.custom_id) for row in outstanding]
)
db.execute(
    "update batch_items set attempts = attempts + 1 where run_id = %s and custom_id = any(%s)",
    (run_id, [r.custom_id for r in outstanding]),
)

Resubmitting the whole file instead would cost roughly 28 times as much and would re-enrich 115,700 products that were already correct, which is not merely wasteful — it is a second opportunity for those rows to come back wrong.

Both halves of that sentence are checkable. Dividing 120,000 by 4,300 gives just under twenty-eight, which is where the multiple comes from, and it is a multiple of the token bill rather than of the wall clock, since either version is one batch submitted once. The second opportunity is the half that costs more than money. Each of those 115,700 rows would be drawn again, independently, from a model that does not return the same answer twice, and the products table carries no column recording which draw a row came from — so a rerun does not refresh the data, it replaces some unknown subset of it with different data of the same quality.

The attempts < 3 bound is what stops a permanently broken input from cycling forever; an item whose request body is malformed in a deterministic way will fail identically on every attempt, and after the third the right move is to mark it abandoned with the detail preserved and let a human decide whether to repair the input or exclude the row altogether.

Not every outcome deserves the same retry treatment. Expired requests should be resubmitted unchanged, since nothing about them was wrong and a fresh window is all they need. Provider errors deserve inspection first: a burst of rate_limit_error on a handful of items is worth retrying immediately, while invalid_request_error on the same item three nights running is a data problem that a retry cannot fix. Schema failures are the ones where blind retry is most tempting and least justified, because a model that returned prose once will often return prose again on the same input, and the fix belongs in the prompt or in a constrained-output setting rather than in the retry loop.

Reconciliation is not schema validation, and it is not idempotency#

The reconciliation table records that a row failed your schema. It has no opinion on what the schema should be or how strict it should be, and having the table does not excuse the absence of a real one — a permissive schema simply moves the damage from schema_failure into succeeded, where nothing is watching for it.

Nor is item-level reconciliation the same as idempotency, the property that applying the same write a second time leaves the data exactly as the first one left it. A table keyed by custom_id tells you which items still need work; it does not stop the loader writing the same successful payload twice if the drain step runs twice, which it will, because the retrieval step in chapter 1 was made safe to re-invoke on purpose. The write has to be an upsert — an insert that updates the existing row instead of failing or duplicating when the key is already present — keyed by the same id for the reconciliation to mean anything.

And none of it helps while the job is still running. Every technique here operates on an output file, and an output file exists only once the provider has stopped. That is fine on a nightly run with hours of slack. It is useless when the run is late and something has to happen before the window closes on its own.

Per-item outcomes are the only honest unit of accounting for a batch, and the cost of adopting them is a table with seven columns and a classification function that never raises. What that buys is the difference between a pipeline whose bad night is a few thousand rows in provider_error and a pipeline whose bad night is a silent corruption discovered a week later by someone looking at something else. The reconciliation table is also what makes every later decision in this topic cheap: the retry set, the coverage alert, and the outstanding set a fallback has to cover are all one query against the same rows.

The case the table cannot reach — a job still running that you no longer want to wait for — is where it earns its keep hardest, and it needs one more mechanism first. Stopping your poller does not stop the job, so getting an output file out of a run in progress means asking the provider to stop, and then working out which items came back. That is the next chapter.