Batch inference: the half of your work that can wait

There is a 50% discount sitting in every major model API that most teams never take. It is not a negotiation, a credit, or a tier you have to qualify for. Anthropic, OpenAI and Google all price their asynchronous endpoints at half their standard rates, and have done for long enough that it is no longer news.

The reason teams leave it there is not ignorance. It is that "asynchronous" sounds like an architecture project, and half of nothing is nothing. Both of those are worth taking seriously, and neither is the interesting question.

The interesting question is narrower: what fraction of your model calls has a person waiting on the other end? For most products the honest answer is lower than the codebase implies, because work that runs synchronously and work that needs to run synchronously drifted apart some time ago and nobody re-examined it.

What the discount actually is

The terms are close enough between providers to summarise in one place, and different enough in the corners that the corners are where the decisions live.

Anthropic's Message Batches API charges 50% of standard prices, takes up to 100,000 requests or 256 MB per batch, and says most batches finish in under an hour with a hard expiry at 24. Requests that expire are not billed. Results stay available for 29 days.

OpenAI's Batch API is also 50%, on every model, with a fixed 24-hour window, up to 50,000 requests or 200 MB per batch, and up to 2,000 batches an hour. If a batch expires, the requests that did finish are still delivered and still billed. It draws from a separate token pool, so batch work does not consume your synchronous rate limits, which for some teams is worth more than the discount.

Google's Batch Mode is 50% of standard cost against a 24-hour target, with a 2 GB limit per input file, and jobs that sit running or pending for more than 48 hours expire with no retrievable results.

Three details in there matter more than the headline. Expiry behaviour differs, and it is the thing that will wake you up. Anthropic's "most batches complete within an hour" is a description of typical behaviour rather than a promise. And the separate rate-limit pool is a real capability that gets ignored because it is not a number on a pricing page.

The discount is not the variable

Here is the arithmetic teams get wrong. The 50% is fixed, so the only thing that moves your bill is how much work you move.

Take 3 million requests a month at 4,000 tokens in and 600 out on Claude Sonnet 5. That is $42,000 a month synchronously. Move a fifth of it and you save $4,200 — a 10% discount on the whole bill, which is real money and nobody's idea of a project. Move 60% and you save $12,600. Move everything that could move and you save $21,000.

The sentence worth writing on the wall: a 50% discount on 20% of the work is a 10% discount. Teams that estimate their batchable share at "some of it" and build the queue anyway are the ones who end up disappointed, and the disappointment is arithmetic rather than execution.

When the plumbing is worth it

Batch endpoints are not a flag you set. You submit a file, poll for completion, retrieve results, match them back to the rows that produced them, and decide what happens to the ones that failed or expired. It is a day or two for something crude and a couple of weeks for something you trust on a Sunday night.

At 250,000 requests a month, 60% batched saves $1,050. Against $6,000 of engineering that is 5.7 months before it earns anything, and it has to be maintained the whole time. At 3 million requests the same build pays back in about two weeks.

This is the same shape as most infrastructure decisions, and it is why the answer is a volume rather than a principle. The honest version for a small product is: not yet, and here is the number at which it changes.

What is genuinely batchable

Sort your model calls by who is waiting. The list is usually shorter on the synchronous side than people expect.

The row worth arguing about is the nightly one. A digest that must be in inboxes by 7am is not latency-sensitive in the usual sense — nobody watches it run — but it is deadline-sensitive, and a 24-hour window cannot promise a 6am finish. Batch it and you have bought a 50% discount on a job that will, on some morning, simply not be there. Either move the submission far enough ahead that a full window still lands before the deadline, or keep the synchronous path for that one job and batch everything else.

Evals are the cleanest case, and usually the largest. A properly sized eval run is thousands of calls that nobody watches, repeated every time a prompt changes. If you only ever batch one workload, batch that one: the saving is immediate and there is no deadline to miss.

:::note kind: money title: Caching and batching stack The discounts multiply rather than compete, so cached input inside a batch is cheap twice over. The catch is that batch requests are processed concurrently, so cache hits are best-effort; Anthropic reports hit rates anywhere from 30% to 98% depending on traffic shape. Anthropic's own guidance is to use the longer cache duration for batches, because a five-minute entry will usually have expired before the request runs. The mechanics are in what prompt caching actually saves. :::

Build the unhappy path first

The failure mode in batch work is not a bad answer. It is silence.

A synchronous call fails loudly, in a request, in front of a stack trace, usually within seconds. A batch fails by not being finished, in a job nobody is looking at, and the first symptom is a downstream table that is short of rows. Everything that makes batch cheap also makes it quiet.

Four things to build before the first production job:

• Reconciliation. Count what you submitted and what came back, and alert on the gap rather than on errors. Expired requests do not raise anything.
• Idempotent resubmission. Assume you will resubmit a partial batch, and make double-processing harmless. The general form of this problem does not change because the queue belongs to someone else.
• A deadline that is not the window. If the work has a time it must be done by, submit at least two windows ahead and check at the halfway point.
• A synchronous fallback for the tail. When a batch expires with 3% outstanding, the cheapest fix is usually to run those few hundred rows at full price rather than wait another day.

What this does not solve

Batch is a discount on tokens. It does nothing about how many tokens you send, which is nearly always the larger lever: an agent loop resending its history, or a tool list riding on every call, will cost more than batching can recover. Work out what a user costs you in a month before deciding that a 50% discount on half your calls is the fix.

It also changes your operational surface. You now have jobs, state, partial results and a queue you do not control, in exchange for money. At high volume that is obviously worth it. At low volume the same engineer could have spent the fortnight on the prompt, and the prompt is where the larger savings usually are.

And it is worth saying that the 50% has been stable for two years across three providers, which is unusual in this market and suggests it reflects genuine scheduling economics rather than a promotion. That stability is what makes it safe to build against. It is also why there is no rush: the discount will still be there next quarter, when your volume may have crossed the line that makes it worth taking.

The short version

• Anthropic, OpenAI and Google all price asynchronous batch work at 50% of standard rates. The discount is not the variable.
• What you save is exactly half of what you move, so estimate your batchable share honestly before building anything.
• On 3 million requests a month, 60% batched saves about $12,600 and pays back a $6,000 build in under a month. On 250,000 it takes 5.7 months.
• Windows are ceilings, not schedules. Deadline-bound work needs two windows of headroom or a synchronous path.
• Expiry behaviour differs by provider: Anthropic does not bill unfinished requests, OpenAI delivers and bills the finished ones, Google returns nothing after 48 hours.
• Caching stacks with batching, at best-effort hit rates of 30% to 98%.
• Build reconciliation, idempotent resubmission and a fallback for the tail before the first production job. Batch failures are silent.