Stop begging the model for JSON. Constrain it with a schema and be done.

Part 1Three runs where the model promised and reneged

Our bill-parsing system needs the same JSON shape every time: vendor name, amount, date, line items, so a pure-code validation layer downstream can pick it up.

Our prompt was explicit: "Return ONLY a JSON object with these keys," followed by every key spelled out.

Three actual runs on the same bill, and the model transcribed the whole document into rambling prose instead, 31 seconds in one run, 56 seconds in another, until it hit the token ceiling, with not a single brace in sight. Our system could only log the result as no-json (no JSON found in the response).

The point worth remembering: this wasn't a bad model. The one we were running had been tuned for transcription work, so the moment it saw a bill image, its instinct was to transcribe everything on the page. Prompt instructions lose to that kind of tuning. And even a more obedient instruct-tuned model is still just being asked. Sooner or later it hits an oddly formatted document and reneges again.

Part 2Move the enforcement to the decoding layer

The fix that actually closed this out was vLLM's structured outputs. Instead of hoping the model picks the right token, you cut the schema-violating options out before the model ever gets to choose. Every token that comes out is walking inside the JSON frame you defined, nothing else. It's the difference between a "please cooperate" sign and rails the train physically cannot leave.

The code on our side turned out shorter than expected:

from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

EXTRACT_SCHEMA = {
    "type": "object",
    "properties": {
        "vendor_name": {"type": "string"},
        "date": {"type": "string"},
        "total": {"type": "string"},
        "line_items": {"type": "array", "items": {
            "type": "object",
            "properties": {"desc": {"type": "string"}, "amount": {"type": "string"}},
            "required": ["desc", "amount"]}},
    },
    "required": ["vendor_name", "date", "total", "line_items"],
}

sampling = SamplingParams(
    temperature=0.0, max_tokens=2500,
    structured_outputs=StructuredOutputsParams(json=EXTRACT_SCHEMA),
)

From that point on, the response parses through json.loads every single time. No more regex hunting for braces.

One thing worth knowing before you use this: the schema should define structure only, don't cram in unnecessary picky constraints. The more complex the grammar (the rule structure compiled from the schema to control tokens), the slower the first-run compile. And for fields that might legitimately be missing, let them be an empty string, that's better than marking every field required and forcing the model to guess a value to fill it in.

Part 3Lesson one: a silent fallback is a time bomb

Early on we wrote what looked like careful code: if the import failed, fall back to the old approach.

try:
    from vllm.sampling_params import GuidedDecodingParams  # old name
    ...
except Exception:
    sampling = SamplingParams(temperature=0.0, max_tokens=2500)  # silent fallback

It looked prudent on the surface. But vLLM 0.24 had already renamed this class to StructuredOutputsParams. The result: the import failed every single time, the system silently fell back to unconstrained mode every single time, with zero error lines to show for it. The bills kept failing exactly the same way as before. It took us a while to chase this down before we spotted the fallback line buried in the log.

The rule we set after that: if a fallback affects task correctness, it must print its status to the log, and someone (or a script) has to confirm after every deploy that the primary mode is actually running. Not erroring is not proof it works.

Part 4Lesson two: keep a truncation backstop

Structured outputs guarantees every token follows the schema, but there's still one way it can break: generation hits the max_tokens ceiling mid-document, and the JSON gets cut off at the end. So we keep a fallback parser that tries to close the brackets before it gives up.

start = text.find("{")
if start >= 0:
    frag = text[start:].rstrip().rstrip(",")
    frag = re.sub(r',\s*"[^"]*$', "", frag)   # drop a dangling key
    for pad in ("", "}", "}}", '"}', '"}]}', "]}"):
        try:
            return json.loads(frag + pad)
        except Exception:
            continue

This layer almost never fires. But on the day an unusually long document shows up, it hands the reviewer partial data instead of a bare error.

Part 5Results after turning it on

The bill that had failed three times in a row parsed clean on the first run after we turned on structured outputs, every key present, amounts matching the source document. Time per document showed no meaningful difference from before. The only added cost was the first-run grammar compile (the token-constraint rules built from your schema), which happens once at model load (cold start).

If your work hands LLM output to code downstream, whether that's extracting data, classifying, or calling an API, here's the short version of what we learned: don't spend time trying to tame the prompt, move the enforcement to the decoding layer from day one, and save your effort for whatever harder problem is actually left. The full system this piece feeds into is on the AI bill scanner page.

Frequently asked questions

How is structured outputs different from just instructing the model in the prompt?

A prompt instruction is asking for cooperation, the model can still pick any token it wants. Structured outputs cuts schema-violating choices out at the decoding layer (the step where the model picks each token), so the model literally cannot answer outside the frame. The result parses every time.

Does structured outputs slow generation down?

In our own measurements, generation time per document showed no meaningful difference from before. There's one added cost: the first-time grammar compile, which happens once at model load. The simpler the schema, the faster that compile.

Can JSON still break with structured outputs turned on?

One case remains: generation hits the max_tokens ceiling mid-document and the JSON cuts off at the end. Two fixes for it: set max_tokens high enough for the longest real document you've seen, and keep a fallback parser that closes the brackets before giving up.

Which vLLM version uses which class name?

Newer versions (roughly 0.11 onward, including the 0.24 we run) use StructuredOutputsParams. The old name GuidedDecodingParams has been removed. If your code has an import-catching fallback, log which mode is actually active so it doesn't fail silently.