Structured Output with LLMs: Schemas, Validation, and Retry Loops

Ask a model for JSON and it will hand you prose with a code fence in it. Ask again and you get valid JSON containing a hallucinated field you never defined. Every team building on LLM APIs discovers this in the same week, usually after an incident where a downstream parser crashed on a stray comma.

Structured output is the discipline of making model responses satisfy a contract your code can rely on, without defensive parsing scattered through the codebase. It requires schema design matching what the model can produce, constrained generation where supported, layered validation, and a repair loop with an explicit failure budget.

Start From the Consumer, Not the Schema

The most common design error is writing a schema first, then finding a use for it. Invert it: find the code that will consume the output and let that determine the shape.

Ask four questions first:

1. Which code reads this? A field nobody reads is pure cost — it consumes tokens, invites generation errors, and enlarges the repair surface.
2. What is the narrowest useful shape? If three fields do the job, five will eventually hold two invented ones.
3. What happens when a field is absent? Optional fields with genuinely optional semantics, not optional fields you treat as required downstream.
4. What must never appear? Extra properties, prose, markdown fences — decide whether these are stripped or rejected, because that choice changes where failure surfaces.

A schema mirroring an existing domain type from your codebase usually beats one invented for the prompt, because it eliminates a translation layer and every bug that layer would host.

Schema Design Rules That Reduce Failures

Flat beats nested, to a depth of two or three. Deep nesting multiplies the ways a generation can go wrong and makes error messages harder to act on. Flatten where the consumer tolerates it.

Prefer enums to free strings for closed sets. An enum is enforced by the grammar and self-documents at the call site. A free string requires the validation you just avoided, plus a mapping layer for every consumer.

Make the model do no arithmetic. Totals, counts, durations, and currency conversions belong in your code. Every one is a place where a plausible number will be wrong.

Name fields for meaning, not layout. Names mirroring a source document's visual structure (section_1, col_b) leak presentation into data and make the schema useless once the source changes.

Descriptions are part of the contract. Every ambiguous field needs units, format, and acceptable range stated. This is the highest-leverage token spend in the prompt, the only channel through which the model learns what you meant.

Avoid over-specifying string formats with rigid regex patterns unless needed. Every added constraint increases failures, and a slightly-off timestamp your parser can normalise beats a rejected response.

Constrained Generation: The Strongest Available Control

When your platform supports grammar- or schema-constrained decoding, use it. This is categorically stronger than prompting, because invalid output becomes structurally impossible rather than statistically unlikely.

The mechanism masks tokens that would violate the schema at each decoding step, so the model can only produce output that parses. This eliminates entire error classes — trailing commas, single quotes, unescaped characters, hallucinated keys — without any retry.

There are caveats. Constrained decoding increases latency, sometimes substantially, because the candidate set per step is smaller and implementations vary in speed. Large schemas permitting many valid orderings also enlarge the search space. And it guarantees the shape is valid, never the content — a required field will be present, but its value may be wrong. Treat it as a syntactic guarantee underneath semantic validation, never a replacement.

The Validation Stack

Design validation as layers, each catching a different failure class:

Layer 1 — Parse. Strict parsing, no lenient fallbacks. If the response is not parseable, that is a fact worth surfacing, not repairing silently. Never use a permissive parser that guesses at intent, because it converts a clear failure into silent data corruption.

Layer 2 — Structural validation. Type checking, required fields, enum membership, array item types — what schema libraries handle well.

Layer 3 — Semantic validation. Rules a schema cannot express: a date range that must be internally consistent, a quantity that must be positive, a category that must exist in your taxonomy, a total that must equal the sum of its parts. Write these as named rules with clear failure messages — the message is what makes repair possible.

Layer 4 — Cross-field and business rules. Checks requiring context outside the response, such as whether a referenced ID exists or the user is entitled to the operation.

Layer separation matters because each layer has a different fix: a parse failure needs a retry, a semantic failure needs targeted correction, a business-rule failure may need escalation.

Repair Loops That Actually Converge

The repair loop is where most implementations waste money, because they retry the same prompt and hope for a different result:

1. Classify the failure before retrying. Only some failures are worth a retry. A parse failure from truncation may succeed with a larger budget; a semantically impossible request will fail identically every time.
2. Return structured error messages to the model, naming the exact field, constraint, and received value. Vague feedback produces vague retries.
3. Escalate constraint specificity across attempts. Attempt one may use the base schema; attempt two may add examples; attempt three may restructure the request. Repeating an identical prompt is not an escalation.
4. Cap attempts explicitly, usually at two or three. Beyond that, return a partial result, decompose the call, or escalate to a human path.
5. Track repair rate by failure class. A rising parse-failure rate after a model update signals that your schema or token budget needs revisiting.
6. Never let an unrepaired response flow downstream as valid. The last-resort failure is a silent fallback to a zero-value object, converting a loud failure into quiet corruption.

Failure Budgets and Degradation

Define in advance what happens when the model cannot produce valid output within budget, in descending order of preference:

• Retry with a narrowed scope, asking for fewer fields or a simpler sub-object, then assembling the result from partial successes. Often the highest-yield strategy for large payloads.
• Decompose the request, generating independent pieces separately so one failure does not invalidate everything.
• Return a typed error, so the caller can decide — a legitimate outcome, since the alternative is inventing data.
• Escalate to a human, for high-value paths where correctness matters more than latency.

Then budget the economics. Repair attempts multiply cost and latency, so track expected calls per success. If your p50 is one call and your p95 is seven, your capacity planning and latency promises are both wrong.

Implementation Checklist

1. Define the schema from the consuming code, not the prompt.
2. Drop every field no consumer reads; flatten nested structures where tolerance allows.
3. Use enums for closed sets; move all computation out of the model.
4. Write a precise description for every field with units, format, and range.
5. Enable constrained generation if the platform offers it.
6. Implement validation in four layers with named semantic rules and clear messages.
7. Classify failures before retrying, escalating constraint specificity rather than repeating the prompt.
8. Cap repair attempts, defining typed-error and human-escalation paths in advance.
9. Instrument repair rate by failure class, plus calls-per-success and cost.
10. Test with truncated, ambiguous, contradictory, and adversarial input.

Testing the Failure Paths

The happy path is not where these systems break. Build a test set targeting failure modes: inputs engineered to exceed the output budget, prompts ambiguous enough that the model must guess, contexts with contradictory information, and inputs that make the schema unsatisfiable. Test the repair loop by injecting known-invalid responses, and confirm exhausted repair yields a typed error rather than a plausible default.

Summary

Reliable structured output comes from designing schemas around their consumers, using enums and flat structures, keeping arithmetic out of the model, and writing precise field descriptions. Constrained generation eliminates syntax failures structurally and should be layered with parse, structural, semantic, and business validation rather than replacing it. Repair loops must classify failures, escalate constraint specificity, cap attempts, and never let unrepaired output masquerade as valid. With an explicit failure budget, structured output becomes a contract you can depend on.

Frequently Asked Questions

Should I use constrained generation or prompting for structured output? Use constrained generation wherever the platform supports it, because it makes invalid syntax structurally impossible rather than merely unlikely, eliminating whole classes of parse failure without retries. The caveat is that it guarantees only valid shape, not correct content: a required field will be present, but its value can still be wrong. Semantic validation remains necessary, and latency varies by implementation.

How many repair attempts are too many? Beyond two or three, retries usually stop converging and start costing money. Repeating an identical failed prompt is wasted spend, so each attempt should escalate constraint specificity, add examples, or narrow or decompose the request. Once attempts are exhausted, switch strategy: return partial results, decompose the call, emit a typed error, or escalate to a human path.

How do I stop invalid outputs from reaching downstream systems? Make invalidity impossible to pass rather than merely unlikely. Use constrained generation for syntax, then run four validation layers — parse, structural, semantic, and business rules — ending in a typed error. The critical rule is that an unrepaired response must never be replaced with zero-value defaults, because that converts a loud failure into silent corruption. Consumers receive a validated object or a typed error, never a placeholder.