“First, don’t ask for JSON in prose and hope. Use the provider’s constrained decoding — attach a schema to the request so invalid tokens are masked at sampling time and the output parses by construction.
That buys you syntactic validity, not semantic validity. The model will happily return a perfectly-shaped object with a date in the wrong format, an enum value it invented, or line items that don’t add up to the total it also gave you. So there are two layers: constrained decoding for shape, then my own validator for meaning.
When validation fails, I retry exactly once with the validation error echoed back — ‘currency must be one of USD, EUR, GBP; you sent US Dollars’. That fixes most of them. If the second attempt fails, it goes to a fallback path or a human. It does not loop.”
The deep dive — diagrams, tradeoff tables, and the follow-up trap
Two gates, two different failures
Constrained decoding gets you shape. It does not get you meaning.
What still goes wrong after the JSON parses
Failure
What it looks like
Guard
Invented enum
"status": "in-progress-ish"
literal enum in the schema; reject and echo the allowed values
Ambiguous unit
"amount": 1250 — cents or dollars?
require an explicit currency and minor-unit field
Doesn’t reconcile
line items sum to 1180, total says 1250
compute the total yourself; never trust the model’s arithmetic
Hallucinated id
an order_id that has never existed
resolve every id against the system of record before acting on it
Short array
3 of 47 line items, and it parses
reconcile the row count against the document; a reply cut off at max_tokens shows in the stop reason
The retry that works
One repair attempt, carrying the specific error. Not a loop, and not a bare retry of the same prompt — that just resamples the same mistake.
def extract(doc, attempt=1, repair_hint=None):
resp = client.messages.create(
model=MODEL, max_tokens=16000,
output_config={"format": {"type": "json_schema",
"schema": INVOICE_SCHEMA}},
messages=build(doc, repair_hint))
if resp.stop_reason in ("refusal", "max_tokens"):
raise Incomplete(doc.id, resp.stop_reason) # may not match the schema
text = "".join(b.text for b in resp.content if b.type == "text")
obj = json.loads(text) # shape already guaranteed
errors = check_business_rules(obj, doc) # enums, units, totals, ids, row count
if not errors:
return obj
if attempt == 2:
return escalate(doc, errors) # human queue; do not keep looping
return extract(doc, attempt + 1, repair_hint=render(errors))
REAL SYSTEM
Invoice extraction, ~40K documents a month. Moving from “respond in JSON” to schema-constrained decoding took hard parse failures from ~4.2% to zero. The business-rule validator then caught another ~6.1% that parsed perfectly — mostly invented currency codes and totals that didn’t match the line items. One repair retry cleared ~81% of those, leaving ~1.2% for a human. The row-count check was added after a month of invoices that were quietly missing their last rows.
FOLLOW-UP TRAP
“If decoding is constrained, why validate at all?” — the grammar only guarantees the shape you asked for. Its most dangerous failure is the one that looks like success: a well-formed object with three of forty-seven rows, because the model stopped listing early. It parses, it validates against the schema, and it is wrong. Reconcile counts against the source, and check the stop reason on every call — a refusal or a max_tokens cut-off may not match the schema at all.
Learn it in full:Structured Output: JSON Every Time — output_config.format, messages.parse(), strict tools, and what a schema cannot say.
