Cancelling a batch job

There are exactly three things that can happen to a batch job you no longer want to wait for, and only one of them is a decision. A batch job here is the asynchronous unit both major providers expose — a whole collection of model requests submitted in one call, run on the provider's schedule, collected later as a results file — and its defining property for this chapter is that it has no idea your process exists. Scaling your poller to zero replicas, killing the pod, or raising an exception in the polling loop all leave the job running, serving requests and accruing charges exactly as before.

This chapter is about the mechanics of walking away deliberately. It sets the three exits against each other, works through what a cancel call actually does on each provider and how long it takes to settle, and shows why the output file a cancelled job leaves behind is the thing that makes a fallback affordable. The running example is a daily product-enrichment job that submits 300,000 rows at 22:00 and normally lands around 02:00, whose per-request identifiers — the custom_id attached to every row at submit time — are already recorded in the reconciliation table built in chapter 3, one row per submitted item with a state column.

Three ways to walk away, and only one of them is a decision#

The differences are not intuitive, which is why they are worth setting side by side rather than describing in turn.

Stopping the poller is the row that costs the most and decides the least. The job runs, you are billed for every request it serves, and you learn nothing about which items are outstanding until it ends — which on a night that has already slipped is hours after the point at which the information would have been useful. It is a decision about your process disguised as a decision about the work.

Letting the window expire is defensible when the deadline is soft, since unstarted requests are returned unbilled and you keep whatever completed. It is a poor fit for a deadline because the terminal state arrives when the provider says so, up to 24 hours after submission, and a fallback cannot start until you know what to run.

Cancelling is the only one of the three that converts "I am not waiting any more" into a fact on the provider's side within a bounded time.

What cancel actually does#

batch = client.messages.batches.cancel(job_id)
print(batch.processing_status)      # canceling
db.execute(
    "update batch_runs set status = 'canceling', cancel_requested_at = now() where provider_job_id = %s",
    (job_id,),
)

The call returns immediately with the job in a transitional state rather than a terminal one, because cancellation is a request to stop scheduling new requests, not a kill switch on the ones already in flight. Anthropic moves the job to canceling and then to ended once the in-flight requests have drained; OpenAI moves it to cancelling and documents that reaching cancelled can take up to ten minutes. Requests that were already being served finish and are billed, which is correct — the tokens were generated — and typically accounts for a few hundred to a few thousand items depending on how wide the provider had the job spread.

The reason it drains rather than stopping is that a generation in progress cannot be un-run. By the time a cancel arrives, some requests have already been handed to a worker and are part-way through producing an answer, and the compute behind the text already emitted has been spent whether or not anyone keeps the result. Killing those requests would throw that work away and leave half-written outputs to be classified; letting them finish costs a bounded amount of extra time and produces ordinary, complete lines in the results file. So the provider stops dispatching immediately and lets what is in flight complete.

The consequence for anyone cancelling under time pressure is that the operation is bounded rather than immediate, and a bound is something a schedule has to carry. That is why the ten-minute figure belongs in the table above rather than in a footnote: a plan that treats cancellation as instantaneous is wrong by exactly the length of the drain, on the one night it is being relied on. The reachable states and the transitions between them are these:

Every path in that diagram terminates at the same place, and that is why it is drawn as a state machine rather than described. ended after a cancel and ended after natural completion are the same state, reached through different histories, and both produce a results file you fetch exactly the way the retrieval step in chapter 1 fetches one — read the job id from your own database, stream the results, store the raw file before parsing. There is no separate cancelled-job endpoint and no special retrieval path. What differs is the composition of the file: alongside the succeeded and errored lines there are now lines with result.type == "canceled" for the requests the provider dropped, in the same position they would have occupied had they run.

So a cancelled job is not a job whose results are lost. It is a job whose results are partial, and partial results are something the machinery already knows how to handle, because the reconciliation table from chapter 3 is populated from the ids you submitted rather than the ids that came back, and canceled was one of its states from the start.

The diff is already in the table#

Draining a cancelled job runs the same code as draining a completed one, and the classification step gains no new branch. Once the drain has settled, the outstanding set is a query:

select custom_id
  from batch_items
 where run_id = %(run_id)s
   and state in ('canceled', 'expired', 'pending', 'provider_error', 'schema_failure');

Everything not in that result set has already been enriched at the batch rate and is sitting in the output file waiting to be loaded. Everything in it is work the fallback has to cover. The gap between those two numbers and the total is the entire value of running the drain, and it is usually much larger than people expect: a job abandoned at four in the morning has generally served most of what it was given. The reason is in what "late" means. A job is abandoned because it has run past the hour it normally lands, which is to say it has already been running for most of a normal duration, and a batch job serves requests steadily rather than saving them all for the end. Late and barely started are therefore different conditions, and the second is rare enough that assuming it is what makes a blind re-drive expensive.

Cancel is not a rollback, and the drain is not instant#

The completed portion of a cancelled job is real, billed, and already generated. There is nothing to refund because there is nothing to take back: the tokens came out of compute that has already run, and the provider can no more un-spend it than you can. Cancelling also does not prevent those results from being written if your loader runs afterwards. If the reason for abandoning the job is that the inputs were wrong rather than that the job was late — a bad prompt template shipped at 22:00, say — then cancelling stops the bleeding and the completed rows still have to be excluded deliberately, which is a state = 'abandoned' update on the reconciliation table rather than anything the provider offers.

Cancellation is also not free of latency, and the ten-minute figure matters when the whole reason you are cancelling is that time is short. The sequence is: issue the cancel, wait for the job to reach a terminal state, drain the output, populate the reconciliation table for 300,000 rows, query the outstanding set, and only then start the fallback. Draining and classifying a few hundred thousand lines is a minute or two of streaming and inserts; the provider's drain is the unpredictable part. Budget ten to fifteen minutes between "we are giving up on this" and "the fallback is issuing its first synchronous request", and measure it on a real cancel rather than assuming, because the first time you find out is not a good night to find out.

There is a temptation to skip the drain entirely — cancel, assume nothing useful came back, re-drive everything — on the grounds that it is simpler and time is short. That is the blind re-drive column of the table above, and it is simpler only until the first night when the outstanding set is 88,000 and the pool cannot absorb 300,000 in the time remaining. The drain is what makes the fallback proportional to the work actually left.

Two smaller cases are worth naming. A job cancelled before the provider has started anything returns an output file where every line is canceled and nothing is billed, which is the clean case and also the least interesting one. And a job that reaches ended between your decision to cancel and the cancel request landing simply ignores the cancel — the API returns the job in its terminal state — so the code path has to treat "cancel failed because the job already finished" as a success rather than as an error. That is safe rather than a fudge, and the state diagram is where you can see why: ended is the state a cancel was trying to reach, so a cancel arriving after it has nothing left to ask for and the job is already in the condition the caller wanted. Writing it as an error costs nothing on an ordinary day and raises on exactly the night when something else has already gone wrong, which is the worst available schedule for a spurious exception.

Abandoning a batch is an operation on the provider's job, and treating it as one is what keeps the fallback proportional. The cancel call converts an intention into a fact within a bounded time, the terminal state it leads to is the same one a natural completion reaches, and the file it leaves behind turns the question "what still needs doing" into a query against rows you wrote at submit time. What that machinery buys, on the workload above, is a night that costs $1,406 and seventeen minutes of synchronous burst instead of $2,943 and an hour the clock did not have.

All of it answers what to do once someone has decided to abandon the job, and says nothing about when that decision should be made. The night in the scenario got that part wrong twice over: 04:30 was chosen because it felt late, and the action taken stopped the watcher rather than the work. A cutoff has to be derived from the deadline and from how long the fallback needs, both of which are measurable, and that arithmetic — along with the slip rate at which a batch path stops being worth its discount — is the next chapter.