“First, don’t ask for JSON in prose and hope. Use the provider’s constrained decoding — attach a schema to the request so invalid tokens are masked at sampling time and the output parses by construction.
That buys you syntactic validity, not semantic validity. The model will happily return a perfectly-shaped object with a date in the wrong format, an enum value it invented, or line items that don’t add up to the total it also gave you. So there are two layers: constrained decoding for shape, then my own validator for meaning.
When validation fails, I retry exactly once with the validation error echoed back — ‘currency must be one of USD, EUR, GBP; you sent US Dollars’. That fixes most of them. If the second attempt fails, it goes to a fallback path or a human. It does not loop.”
The deep dive — diagrams, tradeoff tables, and the follow-up trap
Two gates, two different failures
Constrained decoding gets you shape. It does not get you meaning.
What still goes wrong after the JSON parses
Failure
What it looks like
Guard
Invented enum
"status": "in-progress-ish"
literal enum in the schema; reject and echo the allowed values
Ambiguous unit
"amount": 1250 — cents or dollars?
require an explicit currency and minor-unit field
Doesn’t reconcile
line items sum to 1180, total says 1250
compute the total yourself; never trust the model’s arithmetic
Hallucinated id
an order_id that has never existed
resolve every id against the system of record before acting on it
Silent truncation
3 of 47 line items, and it parses
check the stop reason, not just whether it parsed
The retry that works
One repair attempt, carrying the specific error. Not a loop, and not a bare retry of the same prompt — that just resamples the same mistake.
def extract(doc, attempt=1):
resp = client.messages.create(model=MODEL, tools=[INVOICE_SCHEMA],
messages=build(doc))
if resp.stop_reason == "max_tokens":
raise Truncated(doc.id) # parses fine, is missing rows
obj = resp.content[0].input # shape already guaranteed
errors = check_business_rules(obj) # enums, units, totals, ids
if not errors:
return obj
if attempt == 2:
return escalate(doc, errors) # human queue; do not keep looping
return extract(doc, attempt + 1, repair_hint=render(errors))
REAL SYSTEM
Invoice extraction, ~40K documents a month. Moving from “respond in JSON” to schema-constrained decoding took hard parse failures from ~4.2% to zero. The business-rule validator then caught another ~6.1% that parsed perfectly — mostly invented currency codes and totals that didn’t match the line items. One repair retry cleared ~81% of those, leaving ~1.2% for a human. The truncation check was added after a month of invoices that were quietly missing their last rows.
FOLLOW-UP TRAP
“If decoding is constrained, why validate at all?” — the grammar only guarantees the shape you asked for. Its most dangerous failure is the one that looks like success: hit the token limit mid-array and you can still get a well-formed object with three of forty-seven rows. It parses, it validates against a loose schema, and it is wrong. Check the stop reason on every call.
