Invalid JSON from a model: what the retry actually costs

The first version of every extraction feature has the same line in it. Parse the JSON, and if it throws, ask again. It works, it is three lines, and it survives into production because nothing ever obviously breaks.

Then someone asks what it costs. The answer is surprisingly little, and that is the reason to be suspicious of it. A failure mode cheap enough to ignore on the invoice is one that gets ignored everywhere else too — in the latency graphs, in the error budget, and in the pile of requests that used every retry and failed anyway.

This post prices the retry, then argues that the price was never the point.

The bill is the least interesting number

Take a mid-sized extraction job: 3,000 tokens in, 500 out, 500,000 requests a day on Claude Sonnet 5. If every response parsed first time, that is $5,500 a day.

Now assume 8% of responses come back unusable, and you retry twice before giving up.

The expected number of attempts is 1.086, so you pay 1.086x, an extra $475 a day. At a 25% failure rate — bad enough that you would notice in testing — it is 1.313x, or $1,719.

Neither of those is going to trigger a review. A cost line that grows 8% is indistinguishable from a good week of traffic. This is exactly why teams run for years on a retry loop without ever measuring the failure rate underneath it: the mechanism that would tell them something is wrong is the one they disabled by handling it.

What the retries leave behind

Two costs are not on the invoice.

The first is latency. A request that needs all three attempts takes three times as long — about 7.2 seconds where the median is 2.4. Those are not randomly distributed users. They are the ones whose input was unusual, which correlates with the ones whose request mattered. A p50 that looks fine and a p99 that quietly tripled is the familiar shape of a fan-out tail, arriving through a different door.

The second is the residue.

Two retries at 8% leaves one request in 1,953 broken. At 500,000 a day that is 256 requests that reached the end of the loop with nothing usable. What happens to those is a product decision that nobody made: they became a null, an empty array, a row that silently did not get written, or a support ticket.

And the arithmetic above is the optimistic version, because it assumes each attempt fails independently. They do not. A prompt the model finds genuinely hard — an ambiguous document, an unusual language, a field that is not present in the source — fails the same way three times in a row. The real residue is worse than the formula says, and it is concentrated in exactly the inputs that were already difficult.

Work out your own numbers

Set the failure rate to what you actually measure rather than what you assume. Most teams that instrument this for the first time find a number higher than their estimate, because malformed responses that the retry silently fixed were never counted anywhere.

Constrained decoding is a different kind of answer

A retry loop makes failures rarer. Constrained decoding makes a class of them impossible.

Both Anthropic and OpenAI now ship this. Anthropic's structured outputs and strict tool use guarantee that responses match your schema and that tool inputs validate, which removes missing required fields, wrong types and invented enum values from the list of things that can happen. OpenAI's structured outputs make the same guarantee for its models.

The mechanism is worth understanding because it explains the limitations. The schema is compiled into a grammar that constrains decoding, so tokens that would make the output invalid are simply not available to the sampler. Anthropic's example for strict tool use is a booking tool that wants passengers as an integer: without it you can receive "two" or "2", and with it you receive 2, because the string was never a legal token in that position. That is why the guarantee is absolute rather than statistical. It is also why there is a first-request cost: Anthropic notes extra latency the first time a given schema is used while the grammar compiles, with compiled grammars cached for 24 hours afterwards, and OpenAI says the same of the first request with any new schema. If you generate schemas dynamically per request, you pay that compilation every time, and the feature will look slow for reasons that have nothing to do with the model.

What strict mode does not cover

This is the part that catches teams, because the guarantee is strong enough to be trusted further than it goes.

Constraints are not enforced. Anthropic's documentation lists the JSON Schema features that are unsupported, and the list is longer than most people expect: recursive schemas, minimum, maximum, multipleOf, minLength, maxLength, external $ref URLs, and array constraints beyond minItems of 0 or 1. The important detail is what happens to them. They are not rejected — they are accepted and ignored. A field declared as an integer with minimum: 1 is guaranteed to be an integer. It is not guaranteed to be positive.

So the schema in your repository is stronger than the schema being enforced, and the difference is invisible. Your validator still has to run.

Truncation still happens. If the response hits max_tokens mid-object, the generation stops. Constrained decoding guarantees every token was legal on the way, not that the object was finished. This is the most common remaining failure, and it is worse with strict mode on, because a model steered through a rigid grammar can be more verbose than one writing freely. Check the stop reason, not just the parse.

Refusals still happen. Both providers return a distinguishable refusal rather than schema-conformant data when the model declines. That is a branch in your code, not an error to retry — retrying a refusal three times gets three refusals and costs three times as much.

And validity is not correctness. This is the one no API will ever fix. A perfectly-shaped object with the wrong invoice number is more dangerous than a parse error, because the parse error stopped. There is a documented tendency for constrained generation to take the cheapest schema-legal path, which in practice means optional fields that a free-form response would have filled get quietly omitted. The output is valid. The extraction is worse.

:::note kind: warn title: Mark optional fields carefully If a field is genuinely optional in the data, it is optional. If it is optional only because you were not sure the model would always produce it, you have told the grammar it may skip a field you actually need, and it will. Look at your schema and separate the two cases: it is usually the fastest quality improvement available in this whole area. :::

What to do

The ordering that matters most: measure before you tune. A retry loop is a measurement device you switched off, and the failure rate it is hiding is the input to every other decision on this list. Log the failures, with the stop reason and the first hundred characters of what came back, before you change anything.

After that, strict mode is the single highest-value change, and it is close to free. What it buys you is not a lower failure rate — it is the removal of an entire category, which means the failures that remain are all of the interesting kind. Truncation, refusals and wrongness are worth your attention. A missing closing brace never was.

What the numbers cannot tell you

The failure rate in this calculator is one number standing in for a distribution. Your easy documents fail at nearly zero and your awkward ones fail at 40%, and the average tells you about the bill while hiding the thing that will generate complaints. If you can segment the rate by document type, language or source, do that before optimising anything.

The residual figures assume independence and are therefore floors. And the whole model here treats a failure as binary — parsed or did not — which says nothing about the responses that parsed cleanly and were wrong. On most extraction products that last category is larger than everything discussed above, and the only instrument that finds it is an eval set with enough samples to be worth believing.

The short version

• At 8% failing with two retries you pay 1.086x, about $475 a day on 500,000 requests. The money was never the problem.
• The same configuration triples latency for affected users and leaves 256 requests a day failing outright.
• Retries assume independent attempts. Hard inputs fail the same way every time, so the residue is worse than the formula and concentrated in the difficult cases.
• Constrained decoding removes shape violations completely: missing fields, wrong types, invented enum values.
• It does not enforce numeric or string constraints. Unsupported keywords are ignored rather than rejected, so keep validating.
• Truncation, refusals and confidently wrong answers all survive it. Check the stop reason and branch on refusals instead of retrying them.
• Measure the failure rate before changing anything. The retry loop has been hiding it.