How to Benchmark Local LLMs for Code Generation with a Fixed Test Suite

A fixed test suite turns “this model feels good at code” into a number you can defend. The method is unglamorous: define a set of coding tasks with an automatic pass/fail check, pin the model, runtime, quantization, context length, and sampling settings, then run the same suite repeatedly and record every result. The value is not the score itself — it is that the score stays comparable when you swap a model, change a quantization level, or adjust the harness.

This guide is for a technically curious beginner or intermediate practitioner who already runs a local model and wants a repeatable measurement instead of an impression. It covers suite design, environment pinning, execution-based scoring, and the reporting fields that let someone else reproduce your run.

What a fixed test suite actually fixes

Four things must be frozen before any comparison means anything:

1. The tasks. A stable set of prompts or repositories with a deterministic success condition — usually unit tests, a schema check, or a patch that must apply and pass.
2. The harness. The code that feeds prompts to the model, exposes tools, and decides when a task is finished. The harness is part of the measurement, not a neutral wrapper around it.
3. The runtime. Server software and version, model file and version, quantization, context length, and any flag that changes sampling.
4. The scoring rule. How a run becomes a number: binary pass, pass@k, partial credit, or a judged quality score.

Change any one of these and you have a new experiment. That is the whole discipline, and it matters more for local models than for hosted ones. A 4-bit and an 8-bit build of the same weights are effectively different subjects. Quantization, context length, and the chat template applied by your inference server all change what comes back from an identical prompt. Without a frozen suite, those differences are indistinguishable from luck.

Pin the environment before you score anything

Record at minimum: CPU or GPU and memory, operating system, inference server and version, model identifier and version, quantization, context length, sampling settings, and the number of repetitions. The llm-benchmark project treats these as first-class: it defaults to temperature=0, uses tolerance-based numeric scoring and strict schema validation, and writes versioned JSONL records with task hashes and execution traces so runs can be compared later.

Two environment details are easy to miss. First, context length and KV cache settings change memory use and can change results; the weight file size alone is not the memory requirement. Second, server-side defaults matter — a chat template applied by the server can alter output even when your prompt text is byte-identical.

A minimal starting protocol looks like this:

1. Pick one model build and one inference server; write down both versions.
2. Confirm the endpoint answers a trivial request before running any task.
3. Run one easy task end to end and inspect the raw output.
4. Run the full suite three times with identical settings.
5. Compare the three runs before adding a second model.

Design tasks that produce a verifiable pass or fail

The strongest coding tasks end in an executable check. A task is well formed when a script can answer “did this pass?” without a human reading the code.

Useful task shapes:

• Function-level: implement a function, then run a hidden unit test file against it.
• Repository patch: the model edits a small repository, and the suite applies the patch and runs the project’s tests. llm-benchmark supports repo-patch execution and observed agent-loop execution as distinct scoring modes.
• Workflow trace: the model must call tools in a required order, checked deterministically.
• Schema and format: output must validate against a strict schema — useful for structured code and configuration generation.

Keep the suite small enough to run overnight and broad enough to separate models. Mix difficulty deliberately, and version the task set: when you edit a task, its hash changes and earlier results stop being comparable.

The prompt is part of the task. If your harness wraps each task in a system prompt, a tool description, or a retry instruction, that text belongs in version control next to the test file. Two harnesses running “the same” task with different scaffolding are running different tasks.

Two open instruments you can copy

You do not have to build the harness from scratch. Two MIT-licensed projects published in 2026 show two different philosophies.

harness-benchmarking is a research instrument: a Go implementation of the “Ralph loop” that runs a single fixed prompt against a local LLM and lets the model act through tools until the task verifiably passes its tests, with no human in the loop. Its default subject is Qwen3.6-35B-A3B-oQ6-mtp served by a local oMLX server (an OpenAI-compatible API) on an Apple Silicon Mac, but any OpenAI-compatible endpoint and model work via flags. It keeps one record per run in logs/runs.jsonl and separate code-quality judgments in logs/judgments.jsonl, scored head-to-head against a real Sonnet-medium solution.

llm-benchmark is a broader suite runner. Its README describes 119 tasks across 10 scored categories and 21 scoring modes, with automatic model discovery for LM Studio, Ollama, llama.cpp, vLLM, SGLang, TensorRT-LLM, TGI, KTransformers, and generic OpenAI-compatible backends. It adds crash-safe resume, pass@k, deterministic workflow-trace checks, LLM/rubric judging, result diffs, and a pairwise arena mode with persisted ELO artifacts. Note that the repository’s short GitHub description lists older figures (87 tasks, 16 scoring modes) than the README — check the version you actually pull rather than trusting a summary line.

Both repositories currently show no stars, which is worth stating plainly: these are early, single-maintainer projects. Treat them as readable reference implementations and as sources of method, not as established standards.

Score by execution first, judgment second

Prefer scoring that runs the code. A test that passes or fails is reproducible; a model grading another model’s code is a second measurement with its own error. Both instruments above include judged quality scoring, and both keep it separate from execution results — in a different file or a different mode. Follow that separation.

Use pass@k when a single attempt is noisy: run the task k times and report the fraction of tasks solved at least once, alongside the single-attempt pass rate. Always publish the number of repetitions and the sampling settings next to the number. A pass rate without them is not interpretable.

When you do use an LLM judge, pin the judge model and version, keep the rubric in version control, and report judged scores as a separate axis. Never merge them into one headline number.

Record results so someone else can reproduce them

A useful run record contains, at minimum:

• Date and harness commit hash
• Hardware, operating system, and inference server version
• Model identifier, version, and quantization
• Context length and sampling parameters
• Task set version or hash
• Repetitions and per-task outcomes
• Raw logs, including failures and timeouts
• Token counts and cost when a hosted endpoint is involved

llm-benchmark persists prompt, output, reasoning, and total token counts, plus provider-reported api_cost when available, and supports a spend cap via benchmark.max_api_cost or --max-api-cost USD. That is useful when you run the same suite against a frontier model as a baseline, so the comparison is bounded by a stated budget rather than an open-ended bill.

Common failure modes and how to diagnose them

The score moves but nothing changed. Check sampling settings, server version, and whether the task set was edited. Then check whether the harness changed — a tool-loop change can shift results more than a model swap.

Every task times out. This is usually a harness problem, not a model problem. Verify the endpoint responds, the tool schema is accepted, and the loop has a termination condition. Confirm one trivial task completes before running the full suite.

Results look too good. Suspect contamination. If your tasks come from public repositories or well-known exercises, the model may have seen them. Prefer tasks you wrote, or held-out variants you can verify are not public.

The pass rate is unstable across identical runs. Causes include nonzero temperature, nondeterministic server behavior, and too few tasks. Increase repetitions, set temperature to zero where the task allows, and report a range instead of a single point.

What a fixed suite cannot tell you

A fixed suite measures task completion under one configuration. It does not measure maintainability, readability, or whether the code matches your project’s conventions — harness-benchmarking keeps a separate quality axis for exactly this reason. It does not transfer across quantization levels or hardware without re-running, and a small suite can only distinguish large differences. Report the suite size and the full configuration with any number you publish, and treat cross-machine comparisons as hypotheses until both sides run the same tasks.

Next step

Pick ten tasks you can check automatically, pin one local model and one runtime, and run the suite three times. Write down the environment fields above before you look at the results. If the numbers are stable, expand the task set; if they are not, fix the harness and sampling settings first. If you hit a failure mode that is not listed here, share the symptom, your environment, and the raw log in the community so others can compare against their own runs.