LLM JSON Structured Output: Schema, Zod, and Tool Calling

Why “just ask for JSON” is not enough

Search interest around llm json, openai structured outputs, and mcp tool json schema is not hype noise. Agent frameworks, IDE copilots, and custom RAG pipelines all need machine-readable results. The failure mode is always the same: the model is optimized for fluent language, not for being a deterministic encoder of your TypeScript type.

In shipping systems I have seen three recurring surprises. First, the model wraps valid JSON in markdown fences even when you said “no markdown.” Second, it invents optional fields that your ORM or form layer never expected. Third, under max-token pressure it truncates mid-object soJSON.parse throws, while the log still looks “almost right.” Structured-output APIs reduce the first and second; they do not remove the need for a runtime check.

ByteJSON’s angle is practical: you already generate and validate JSON and schemas in the browser. Use that loop to design the contract, then enforce it in code next to the model call.

The four layers of structured LLM JSON

Production stacks combine the bottom two rows: constrained generation when the provider supports it, plus local validation. Skipping the last row is how “works in the playground” turns into a 2 a.m. incident when a new model version starts emitting slightly different enums.

JSON Schema and provider structured outputs

Provider structured-output features (OpenAI’s structured outputs is the most searched example) accept a schema and constrain the model’s tokens so the reply is more likely to be valid JSON matching that schema. Keyword clusters like json schema openai and openai structured outputs pydantic show the same developer intent across TypeScript and Python shops.

Start from a realistic sample payload—not an aspirational one. Paste it into the JSON Schema Generator, then tighten types by hand (enums, required fields, additionalProperties). Over-wide schemas (“everything is string | null”) reintroduce the chaos you were trying to leave.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["intent", "confidence", "entities"],
  "properties": {
    "intent": {
      "type": "string",
      "enum": ["search", "create", "update", "delete", "unknown"]
    },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
    "entities": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["type", "value"],
        "properties": {
          "type": { "type": "string" },
          "value": { "type": "string", "minLength": 1 }
        }
      }
    }
  }
}

Anti-intuitive detail: strict mode often rejects useful JSON Schema features (or supports only a subset). If generation fails empty or the API returns a schema error, simplify: fewer oneOf branches, no exotic formats, closed enums. Then re-check the sample against the JSON Schema Validator.

Validate LLM JSON with Zod (runnable pattern)

For TypeScript services, Zod is the usual runtime gate. Generate a first draft with JSON to Zod, then edit. The following pattern is intentionally boring: parse text → validate → branch on failure. Boring is what you want at the model boundary.

import { z } from "zod";

const ExtractionSchema = z.object({
  intent: z.enum(["search", "create", "update", "delete", "unknown"]),
  confidence: z.number().min(0).max(1),
  entities: z.array(
    z.object({
      type: z.string().min(1),
      value: z.string().min(1),
    })
  ),
});

export type Extraction = z.infer<typeof ExtractionSchema>;

/** Strip common model wrappers before JSON.parse */
export function extractJsonText(raw: string): string {
  const trimmed = raw.trim();
  const fenced = trimmed.match(/```(?:json)?\s*([\s\S]*?)\s*```/i);
  if (fenced?.[1]) return fenced[1].trim();
  const start = trimmed.indexOf("{");
  const end = trimmed.lastIndexOf("}");
  if (start >= 0 && end > start) return trimmed.slice(start, end + 1);
  return trimmed;
}

export function parseModelJson(raw: string): Extraction {
  const text = extractJsonText(raw);
  let data: unknown;
  try {
    data = JSON.parse(text);
  } catch {
    throw new Error("Model returned non-JSON text");
  }
  return ExtractionSchema.parse(data);
}

// Example usage after any chat/completions call:
// const extraction = parseModelJson(completionText);

Expert preference: prefer .parse (throw) at the service edge and map Zod issues to a controlled retry prompt (“return only the JSON object matching …”). Prefer .safeParse when you must keep the request thread alive and show field-level UI errors.

Related deep dive without the LLM angle: Zod schema generation from JSON. The difference here is the threat model: API partners are usually well-formed; models are not.

Agent tool calling and MCP tool JSON

Tool calling is structured JSON under another name. The model emits a function name plus arguments; your runtime executes the side effect. That is why agent tool schema and mcp tool json schema show up in autocomplete: teams are standardizing how tools declare inputs.

In MCP-style designs, a tool description typically includes a JSON Schema for arguments. Clients list tools, models pick one, and hosts validate arguments before running. If you skip validation, a hallucinated path or unbounded string becomes a security bug, not a UX bug.

{
  "name": "create_issue",
  "description": "Create a tracked issue in the project board",
  "inputSchema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["title", "priority"],
    "properties": {
      "title": { "type": "string", "minLength": 3, "maxLength": 120 },
      "priority": { "type": "string", "enum": ["low", "medium", "high"] },
      "labels": {
        "type": "array",
        "items": { "type": "string" },
        "maxItems": 8
      }
    }
  }
}

Practical rule: keep tool argument schemas smaller than your domain model. Tools are verbs with a few parameters, not full database rows. Large schemas increase invalid generations and make prompt cache less effective.

Agent skills, prompts, and JSON contracts

“Skill” packs (system prompts plus tool lists plus optional schemas) are trending because they package repeatable agent behavior. Whatever brand name you use—skill, playbook, agent config—the durable piece is still data: which tools exist, what JSON they accept, and what success looks like.

When you author a skill for an IDE agent or an internal bot, write the tool JSON first, then the prose instructions. Teams that write a long natural-language skill file without schemas rediscover the same bugs the industry already solved with OpenAPI and JSON Schema. If your skill needs a structured handoff between steps, define that handoff as JSON Schema and validate it the same way you validate LLM final answers.

// Minimal skill-oriented handoff object between agent steps
{
  "step": "research",
  "status": "done",
  "artifacts": [
    { "kind": "url", "value": "https://example.test/docs" }
  ],
  "next_hint": "draft_outline"
}

GEO note: models and answer engines prefer pages that define terms, show valid/invalid examples, and state failure modes. That is why this article includes tables and explicit “when not to” guidance—the same structure helps both human readers and retrieval systems.

Failure modes, repairs, and retries

For near-miss syntax, try the JSON Repair tool during debugging, then decide whether automated repair is acceptable in production. Repairing trailing commas is usually safe. “Repairing” by guessing missing required fields is not—that is inventing data.

async function completeWithJsonRetry(callModel, userPrompt, maxAttempts = 2) {
  let lastError = null;
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const prompt =
      attempt === 1
        ? userPrompt
        : userPrompt +
          "\n\nPrevious output was invalid JSON or failed schema validation: " +
          String(lastError) +
          ". Reply with a single JSON object only.";
    const raw = await callModel(prompt);
    try {
      return parseModelJson(raw);
    } catch (err) {
      lastError = err;
    }
  }
  throw lastError;
}

When you should not force structured JSON

• Open-ended teaching, essays, or design brainstorms where structure kills usefulness.
• Multi-document answers that need citations and prose more than fields.
• Schemas larger than the model can hold in context—split into tool steps instead.
• High-stakes factual claims without retrieval; structure does not create truth.

A useful split: use free-form generation to draft, then a second structured pass to extract fields you will store. Two cheap calls with clear contracts beat one overloaded prompt that tries to be essay and database row at once.