Structured Output: How to Force AI to Return JSON

👁 117 reads

Structured output is an API mode where the model must respond with JSON that matches your schema: exactly those fields, types, and allowed values. Without it, you ask “respond in JSON” and hope for the best, while your Python parser crashes every so often. Below, we show both approaches with real runs, schema generation from Pydantic, result validation, and pitfalls we encountered. This is part of the AI API from scratch path. If the response comes from a closed list, an alternative is AI decision models, which don’t generate text at all.

The Problem: “Respond in JSON” Isn’t Enough

First, we checked what happens when you simply ask for JSON in words, without any mode:

"""For comparison: asking for JSON only in words, without response_format. Do NOT do this in production."""
import os, sys, json
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
MODEL = sys.argv[1] if len(sys.argv) > 1 else "mistralai/mistral-small-2603"

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content":
               "Give me the three largest cities in Poland with population in JSON format. "
               "Fields: city, population."}],
    temperature=0.7,
)
text = r.choices[0].message.content
print("Model response:n" + text)
try:
    print("njson.loads OK:", json.loads(text))
except json.JSONDecodeError as e:
    print(f"njson.loads DOESN'T WORK: {e}")

Model response:
[
  {"city": "Warsaw", "population": 1860281},
  {"city": "Krakow", "population": 804237},
  {"city": "Wroclaw", "population": 674132}
]

json.loads DOESN'T WORK: Expecting value: line 1 column 1 (char 0)

The model did exactly what we asked, just “human-style”: it wrapped the JSON in a code block with three backticks, because that’s how it formats chat responses. For a human it’s readable, for json.loads - an error at the first character. Other typical bugs from our practice: an introductory sentence before the JSON (“Here’s the data:”), field names in the wrong language, numbers as text (“1 860 281”), missing fields. We dissected five such cases in the article Model promised JSON, parser returned null.

There are three levels of format enforcement:

1. Request in prompt - usually works, but no guarantee (as above).
2. JSON mode ({"type": "json_object"}) - model returns syntactically valid JSON, but with any fields.
3. Structured output with schema (json_schema + strict: true) - valid JSON with exactly your fields and types. This is what we use.

Structured Output Step by Step: Invoice Data

Task: extract number, tax ID, date, line items, and amount from invoice text. We describe the schema with a Pydantic class, and Pydantic converts it to JSON Schema - we don’t write it manually. After the response, the same class validates the result.

"""Structured output: model must return JSON matching the schema, and Pydantic validates it."""
import os, sys
from dotenv import load_dotenv
from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field, ValidationError

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
MODEL = sys.argv[1] if len(sys.argv) > 1 else os.getenv("MODEL", "openai/gpt-6-luna")


class LineItem(BaseModel):
    model_config = ConfigDict(extra="forbid")  # in schema: additionalProperties=false
    name: str
    quantity: int
    net_price_usd: float


class Invoice(BaseModel):
    model_config = ConfigDict(extra="forbid")
    number: str
    seller_tax_id: str = Field(description="10 digits without dashes")
    issue_date: str = Field(description="format YYYY-MM-DD")
    line_items: list[LineItem]
    total_gross_usd: float


TEXT = """Invoice no. FV/09/2026/114 dated September 30, 2026.
Seller: Coffee and Code LLC, Tax ID 525-281-44-07.
1) AI API Training, 1 pc., $450.00 net
2) Online Consultation, 3 hrs. at $62.50 net
Total payable: $784.13 gross (VAT 23%)."""

# JSON schema generated by Pydantic - no need to write it manually.
schema = Invoice.model_json_schema()

r = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You extract data from invoices. You return only JSON."},
        {"role": "user", "content": TEXT},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "strict": True, "schema": schema},
    },
    # only providers that support json_schema
    extra_body={"provider": {"require_parameters": True}},
)

raw = r.choices[0].message.content
print("Raw response:n" + raw)

try:
    invoice = Invoice.model_validate_json(raw)
except ValidationError as e:
    sys.exit(f"JSON failed validation:n{e}")

# Type validation isn't everything - we also check data sense
net_sum = sum(item.quantity * item.net_price_usd for item in invoice.line_items)
print("nAfter validation:")
print(" number:", invoice.number)
print(" Tax ID:", invoice.seller_tax_id, "(10 digits)" if invoice.seller_tax_id.isdigit() and len(invoice.seller_tax_id) == 10 else "(BAD FORMAT)")
print(" date:", invoice.issue_date)
print(f" net sum from line items: ${net_sum:.2f}")
print(f" gross from invoice: ${invoice.total_gross_usd:.2f}")
print(" matches with VAT 23%:", round(net_sum * 1.23, 2) == round(invoice.total_gross_usd, 2))

Result for GPT-6 Luna:

Raw response:
{"number":"FV/09/2026/114","seller_tax_id":"5252814407","issue_date":"2026-09-30","line_items":[{"name":"AI API Training","quantity":1,"net_price_usd":450.00},{"name":"Online Consultation","quantity":3,"net_price_usd":62.50}],"total_gross_usd":784.13}

After validation:
 number: FV/09/2026/114
 Tax ID: 5252814407 (10 digits)
 date: 2026-09-30
 net sum from line items: $637.50
 gross from invoice: $784.13
 matches with VAT 23%: True

The model converted “525-281-44-07” to 10 digits on its own, “September 30, 2026” to 2026-09-30, split “3 hrs. at $62.50” into quantity and unit price, and transcribed amounts with commas to numbers with periods. The same script with Gemini 3.8 Flash (python 05_structured_output.py google/gemini-3.8-flash) gave identical data; the only difference was in number notation (450.0 instead of 450.00), which doesn’t matter for Python.

Note the two levels of checking:

• Schema validation (Invoice.model_validate_json) - whether all fields are present and have correct types. With strict: true it should always pass, but we keep it because not every provider enforces the schema the same way.
• Sense validation - whether the tax ID has 10 digits, whether the sum of line items with VAT matches the total amount. The schema won’t check this, and this is where model mistakes show up.

Shorter: .parse() and Pydantic Class

The openai library has a .parse() method that creates the schema from the class and immediately returns a ready object instead of text. It also works through OpenRouter. Example: customer review analysis with a field of limited values (Enum), a list, and a boolean value.

"""Shorter: the .parse() method from the openai library creates the schema from the Pydantic class and validates the result itself."""
import os, sys
from enum import Enum
from dotenv import load_dotenv
from openai import OpenAI
from pydantic import BaseModel

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
MODEL = sys.argv[1] if len(sys.argv) > 1 else os.getenv("MODEL", "openai/gpt-6-luna")


class Sentiment(str, Enum):
    positive = "positive"
    neutral = "neutral"
    negative = "negative"


class Review(BaseModel):
    sentiment: Sentiment
    product: str
    issues: list[str]
    should_reply: bool


REVIEWS = [
    "Courier threw the package under the gate, and the coffee maker has a cracked water tank. I demand a refund!",
    "Headphones sound great, battery lasts 3 days. Recommend.",
    "Lamp OK, though the cable could be longer.",
]

for text in REVIEWS:
    r = client.chat.completions.parse(
        model=MODEL,
        messages=[
            {"role": "system", "content": "You analyze customer reviews for an online store."},
            {"role": "user", "content": text},
        ],
        response_format=Review,
        extra_body={"provider": {"require_parameters": True}},
    )
    review = r.choices[0].message.parsed  # ready Review object, not text
    print(f"{review.sentiment.value:10} | {review.product:22} | reply: {'yes' if review.should_reply else 'no':3} | {review.issues}")

GPT-6 Luna:

negative   | coffee maker           | reply: yes | ['Courier threw package under gate', 'Cracked water tank', 'Customer demands refund']
positive   | headphones             | reply: no  | []
positive   | lamp                   | reply: no  | ['Cable could be longer.']

Claude Haiku 4.5 (same script, different model in argument):

negative   | coffee maker           | reply: yes | ['damage during delivery', 'cracked tank', 'courier negligence']
positive   | Headphones             | reply: no  | []
neutral    | Lamp                   | reply: no  | ['cable too short']

Format in both cases is perfect: sentiment always one of three allowed values, types match. But for the third review (“Lamp OK, though the cable could be longer”) Luna considered it positive, while Haiku considered it neutral. This is exactly the limit of structured output: the schema guarantees format, not judgment. For classification, prepare 20-50 examples with correct answers and check which model and which category description gives the fewest mistakes.

Pitfalls You’ll Encounter

1. In strict mode, every field must be required, and additionalProperties set to false. Pydantic adds this when you set model_config = ConfigDict(extra="forbid") in the class, and .parse() does it automatically. You write an optional field as str | None - the model will then enter null.
2. Not every provider supports schema. On OpenRouter without "provider": {"require_parameters": true} the request may reach a provider that ignores response_format, and you’ll get plain text.
3. Truncated response means broken JSON. When the model hits max_tokens (finish_reason=length), JSON cuts off mid-way. In reasoning models the limit includes thinking, so set it with a large margin.
4. Refusal. When the model refuses to respond (e.g., for safety reasons), instead of data you get a refusal field. Check it before parsing.
5. Overly complex schema. Providers have limits on nesting depth and number of fields. With dozens of fields, it’s better to split the task into two requests.
6. Field descriptions help. Field(description="format YYYY-MM-DD") goes into the schema and the model reads it. This is cheaper than adding instructions to the prompt.

Structured Output vs Function Calling

Function calling (tool use) uses the same schema mechanism, but for a different purpose. Structured output is “respond in this format”. Function calling is “if you need to, ask me to execute this function with these arguments” - this is what AI agents are built on. Once you master schemas here, transitioning to tools will be simple: Function calling step by step, and then AI agent in Python. How agents work and where they break, we described in the article Agent AI: what it is.

What We Use Structured Output For

• extracting data from emails, invoices, listings, and resumes to spreadsheets or databases;
• classifying tickets (topic, urgency, department) in automations, e.g., in n8n;
• generating content in a fixed layout (title, lead, bullet list) for page templates;
• evaluating texts according to criteria with a numerical scale;
• every automation step after which the next program must read the result - we describe them in the article AI automation: where to start.

Frequently Asked Questions

What is structured output?

An API mode where you provide a JSON schema, and the model generates a response that must match it: same fields, types, and allowed values. The provider enforces this during generation, not just after the fact.

How does JSON mode differ from structured output?

JSON mode only guarantees syntactically valid JSON, without field control. Structured output with schema and strict: true guarantees exactly your structure.

Does Claude support structured output?

Yes. In the Anthropic API, this is done through the output format parameter with JSON schema, and through OpenRouter the same response_format works as for OpenAI - we tested this on Claude Haiku 4.5.

Can I use structured output in local models?

Yes, Ollama accepts JSON schema in the format parameter, and llama.cpp has output constraint by grammar. Quality depends on model size.

Why am I getting plain text despite response_format?

The model or provider doesn’t support this parameter and silently ignored it. On OpenRouter add "provider": {"require_parameters": true} - then instead of silent ignoring you’ll get an error or a provider that supports the schema.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Sources and Verification Date

We ran the scripts on October 4, 2026 on Windows 11, Python 3.13.15, openai 3.24.0, pydantic 2.13.5, through OpenRouter. Cost of all requests from this article: approximately $0.003.

• OpenRouter - structured outputs
• OpenAI - structured outputs
• Anthropic - structured outputs
• Google - structured output in Gemini API
• Pydantic - JSON Schema

Kinetyra