Testing a local model for JSON schema compliance means measuring two separate things: whether the model produces parseable JSON at all, and whether that JSON satisfies every constraint in your schema. The EleutherAI lm-evaluation-harness ships a JSONSchema Bench that splits these into two binary metrics — json_validity and schema_compliance — and you can reuse the same definitions against your own schemas with a short Python script. This guide is for practitioners running models behind a local inference server who want a repeatable number instead of a one-off impression.
Two metrics, not one
The JSONSchema Bench documentation defines the evaluation with exactly two binary metrics. json_validity checks whether the generated output is valid JSON, using the Python json package; 1 means valid, 0 means invalid. schema_compliance checks whether the generated output complies with the provided JSON schema, using the jsonschema package; again 1 or 0.
Keeping them separate is what makes the result diagnostic. A model that wraps every answer in a Markdown fence scores 0 on json_validity even if the object inside the fence is perfect. A model that returns clean JSON but puts a string where your schema requires an integer scores 1 on json_validity and 0 on schema_compliance. Collapsing the two into a single pass/fail number hides which of those two problems you actually have.
Prerequisites
- A Python environment. The benchmark tasks depend on the
jsonschemalibrary. - The
formatextra:pip install jsonschema[format]. The documentation states this extra is required to support theformatkeyword in JSON Schema, which is used in the tasks. Without it, schemas that rely onformatvalues such as date-time or email will not be checked as intended. - A local inference server that exposes a chat or completions endpoint — Ollama,
llama.cpp‘s server, vLLM, or a similar runtime. Structured-output enforcement on local servers is commonly handled through JSON schemas and GBNF grammars. - Enough context window for the task you choose. The harness README estimates 2K tokens for the
easytask, 3K formedium, and 10K forhard, noting the exact figure varies with the tokenizer.
Route 1: run the standard JSONSchema Bench
The harness exposes three tasks — jsonschema_bench_easy, jsonschema_bench_medium, and jsonschema_bench_hard — corresponding to the github_easy, github_medium, and github_hard splits of the original paper. The tag jsonschema_bench runs all three.
A typical invocation against a local server looks like the template below. Treat it as a starting shape and confirm the exact flags against the current README, since model-backend argument names change between harness releases.
lm_eval --model local-completions
--model_args base_url=http://localhost:11434/v1/completions,model=your-model
--tasks jsonschema_bench
--batch_size 1Two operational notes from the documentation are worth carrying into your run. First, if memory is tight you can pass --max_length to shrink the context window, but this truncates the schema and will lower measured performance — so a reduced-length run is not comparable to a full-length one. Second, the three splits have different context requirements, which means a model that fits easy may not fit hard without truncation.
Route 2: test your own schemas
Benchmark schemas are drawn from GitHub and will not look like the schemas your application actually uses. A short local harness closes that gap. The script below is an illustrative example, not a measured result: it sends one prompt per call, applies the same two metrics the benchmark uses, and records which of the two failed.
import json
import jsonschema
import requests
SCHEMA = {
"type": "object",
"properties": {
"title": {"type": "string"},
"priority": {"type": "integer", "minimum": 1, "maximum": 5}
},
"required": ["title", "priority"],
"additionalProperties": False
}
PROMPT = "Return only JSON matching this schema: " + json.dumps(SCHEMA)
def run_once(url, model):
r = requests.post(url, json={
"model": model,
"messages": [{"role": "user", "content": PROMPT}],
"temperature": 0,
"seed": 1
}, timeout=120)
r.raise_for_status()
return r.json()["choices"][0]["message"]["content"]
def score(text):
try:
obj = json.loads(text)
except json.JSONDecodeError:
return {"json_validity": 0, "schema_compliance": 0}
try:
jsonschema.validate(obj, SCHEMA)
return {"json_validity": 1, "schema_compliance": 1}
except jsonschema.ValidationError:
return {"json_validity": 1, "schema_compliance": 0}Three practical points about this harness. Use a validator instance with a format checker if your schema uses the format keyword, otherwise format constraints are silently ignored. Run each schema several times rather than once, because a single sample at any temperature tells you very little. And store the raw model output alongside the score — the failing text is what tells you whether the fix belongs in the prompt, the sampler, or the model choice.
Constrained decoding changes what you are measuring
Many local servers can constrain generation directly, using JSON schema or GBNF grammars, so the sampler can only emit tokens that keep the output on a valid path. This is a different experiment from unconstrained generation. With grammar-constrained decoding, json_validity should be near-perfect by construction, and the interesting question moves to schema_compliance and to semantics — whether the values inside the structure are actually correct.
Two cautions follow. First, a grammar guarantees shape, not meaning: a constrained model can emit a perfectly formed object with a wrong priority value. Second, if you report a compliance score produced under constrained decoding, say so, because it measures the server-plus-grammar stack rather than the model’s native ability to follow a schema. Comparing a constrained run against someone else’s unconstrained run is not a like-for-like comparison.
What to record for a reproducible result
A JSON schema compliance number is only useful if someone else can reproduce it. Record, at minimum:
- Hardware and operating system.
- Runtime and version, plus the inference server and its version.
- Model name, version or revision, and quantization.
- Context length setting and maximum output tokens.
- Sampling settings: temperature, top-p, top-k, seed, and repetition count.
- The exact prompt template, including the system message.
- The schemas used, and whether they came from the benchmark or from your application.
- The metric definitions — explicitly, that
json_validityandschema_complianceare binary and computed separately. - Whether constrained decoding was enabled.
Without the last item in particular, a high score is ambiguous: it could reflect model capability or a grammar that made failure impossible.
Common failure modes and what they indicate
| Observed output | json_validity | schema_compliance | Likely cause |
|---|---|---|---|
| Markdown fence or prose preamble around the object | 0 | 0 | Prompt does not forbid surrounding text; parser does not strip fences |
| Truncated object, missing closing brace | 0 | 0 | Output token limit too low, or context exhausted |
| Trailing commas or unquoted keys | 0 | 0 | Model emitting JSON-like text rather than JSON |
| Valid JSON, wrong types or missing required field | 1 | 0 | Schema not respected; prompt or grammar does not carry constraints |
| Valid JSON, extra properties present | 1 | 0 | additionalProperties set to false but not enforced |
Valid JSON, format value wrong | 1 | 0 | jsonschema installed without the format extra, or format checker not attached |
| Schema itself truncated | varies | 0 or misleading | Context too small for the schema; see the harness guidance on --max_length |
Reading the result
The two metrics form a simple diagnostic grid. High json_validity with low schema_compliance points at instruction-following and constraint enforcement: the model can write JSON but is not honouring your structure. Low json_validity points at generation and parsing: the model is producing something that is not JSON, or your extraction step is not isolating the JSON. Both low usually means the task setup itself is wrong — the prompt, the context budget, or the endpoint.
When schema_compliance is high but the application still misbehaves, the failure has moved past schema validation into semantics. Schema compliance is a structural guarantee; it says nothing about whether the extracted value is factually right.
Limitations to keep in view
The benchmark splits are derived from GitHub schemas, so a strong score there does not transfer automatically to a domain-specific schema with unusual nesting, large enums, or heavy use of format. Sample size matters too: a handful of prompts at temperature 0 gives a reproducible but narrow estimate, and a model that passes ten schemas may still fail the eleventh. Finally, results are only comparable within a fixed configuration — changing quantization, context length, or the inference server invalidates the comparison.
A practical next step
Collect ten to twenty schemas that your application actually uses, run each one several times through your local server, and score every output with the two-metric script above. Keep the raw outputs. That small dataset will tell you more about whether a given local model is usable for structured output in your workflow than any aggregate benchmark number, and it gives you a baseline to re-run after changing the prompt, enabling constrained decoding, or switching models.
If your results differ from what others report, the useful details to share are the model and quantization, the runtime and server version, the context and sampling settings, whether a grammar was active, and the exact schema that failed.
FAQ
What is the difference between valid JSON and schema compliance?
Valid JSON only means the text parses. Schema compliance means the parsed object also satisfies the constraints in your schema — required fields, types, ranges, enums, and additionalProperties rules. The JSONSchema Bench measures these as two separate binary metrics so you can tell which one failed.
Does a GBNF grammar or constrained decoding guarantee schema compliance?
No. Constrained decoding restricts generation to a permitted grammar, which makes syntactically valid output very likely, but it does not guarantee that every semantic constraint in your schema is satisfied, and it says nothing about whether the values are correct. Validate the output against the schema regardless.
Why does schema_compliance fail even though the JSON looks right?
Common causes are type mismatches, missing required properties, extra properties when additionalProperties is false, enum violations, and format values that were never checked because jsonschema was installed without the format extra. Inspect the ValidationError message to identify the specific constraint.
Do benchmark scores predict performance on my own schemas?
Not directly. The JSONSchema Bench splits are derived from GitHub schemas, so they are useful as a general signal but do not represent your domain's nesting depth, enum sizes, or format usage. Run a small harness against your real schemas before deciding.