How to Compare Local Language Models with Reproducible Tests

To compare local language models fairly, treat the comparison as a controlled experiment rather than a prompt duel. A reproducible test pins the model, runtime, hardware, prompt template, context settings, and sampling parameters; defines the decision the test must support; runs the same tasks across candidates; and records quality, latency, memory, and failure rates. This guide explains how to do that for local models, from a first small experiment to a durable evaluation workflow. It is written for technically curious readers who are moving from casual local inference to practical implementation, and it focuses on local checkpoints, though the same discipline applies when a local model is compared with a hosted API.

The boundary is important: this is not a catalogue of leaderboard scores, and it does not claim that one model wins universally. The goal is evidence for routing decisions, such as which local checkpoint is reliable enough for private summarization, which one fits a 24 GB GPU, or whether a quantized model is a credible substitute for a metered API on a specific workload. Hermanity’s guide to local LLM benchmarking makes this point directly: a useful benchmark must answer more than how fast one prompt ran, and the goal is not to crown a universal winner but to build evidence for routing decisions. The same guide recommends recording model family, exact checkpoint, quantization, runtime version, prompt template, context length, KV-cache configuration, sampling parameters, GPU model, memory, power limit, and requested and served model identifiers. Those fields are the backbone of reproducibility.

What reproducible local model testing actually means

Reproducibility means that another person, or you six months later, can rerun the same procedure and obtain results that are comparable enough to support the same conclusion. It does not mean that every token will be identical. Local inference involves floating-point arithmetic, GPU scheduling, runtime kernels, batching, and sampling, so exact output can vary even with the same seed. A reproducible test therefore records the conditions that matter and reports the variation that remains.

This is different from three common practices. The first is ad hoc prompt testing: trying a few prompts in a chat interface and forming an impression. That can be useful for exploration, but it is not a comparison because the prompt, context, and sampling settings change between candidates. The second is relying on public leaderboards. Leaderboards can screen candidates, but they rarely match your task, your prompt format, your quantization, your hardware, or your privacy constraints. The third is reading a model card and assuming the reported benchmark scores will transfer. Model cards are valuable evidence, but they are produced under different conditions and may not reflect your runtime or your data.

The arXiv paper Lessons from the Trenches on Reproducible Evaluation of Language Models describes the broader problem: effective evaluation remains an open challenge, and researchers face sensitivity to evaluation setup, difficulty comparing methods, and a lack of reproducibility and transparency. Those issues become practical when you are choosing a local model for a workflow. A model that scores well on a general benchmark may fail at structured JSON, long-context summarization, or code debugging in your environment. A reproducible test narrows the gap between a general claim and your specific decision.

The essential components work together as a chain. The decision defines what matters. The task set represents that decision with concrete inputs. The model identity and environment define what is being tested. The harness runs the tasks under controlled conditions. The metrics turn raw outputs into evidence. The log preserves the evidence so the comparison can be audited or repeated. If any link is missing, the comparison becomes an anecdote.

Start with the decision, not the model

A benchmark is only meaningful relative to a use case. Before downloading checkpoints or writing scripts, write the decision the test must support. Hermanity’s guide lists common questions that make good starting points: Which local model is reliable enough for private summarization? Which checkpoint is best for code generation on one 24 GB GPU? Does a larger quantized model outperform a smaller higher-precision model? How much latency is added by a long context? Where should an agent route structured JSON, debugging, or creative work? Is the local model a credible substitute for a metered API on this workload?

Each question implies different tasks and metrics. Private summarization needs factual consistency, privacy, and acceptable latency. Code generation needs correctness, syntax validity, and sometimes execution results. Structured JSON needs schema compliance and low failure rates. Long-context work needs memory measurements and tests that actually use the long context. A local substitute for an API needs cost, latency, privacy, and reliability comparisons, not just tokens per second.

Write acceptance criteria in advance. For example: the model must produce valid JSON on at least a defined share of test cases, summarize without inventing facts, run within a memory ceiling, and return a response within a latency budget. The exact numbers depend on your workflow. The important part is that you decide what counts as good enough before you see the results. Without pre-committed criteria, it is easy to rationalize the model you already prefer.

If the decision is not clear, start smaller. Pick one workflow, such as summarizing WordPress drafts or classifying support messages, and compare two models on that single workflow. A narrow, well-defined test is more useful than a broad, vague one.

Pin the complete model identity and environment

At Qwen or a 30B model is not enough. The same model family can have multiple checkpoints, and the same checkpoint can be quantized in different ways. The runtime can also change behavior. Hermanity’s guide recommends recording a set of fields that prevent a later rerun from silently testing different bytes under the same name.

Record these fields in a configuration file and in the results log. A simple JSON configuration is enough for a first experiment. The following example is illustrative, not a tested environment. Replace every value with the one you actually use.

{
  'experiment': 'summarization-comparison',
  'model_id': 'example/model-checkpoint',
  'quantization': 'Q4_K_M',
  'runtime': 'local-runtime',
  'runtime_version': 'pinned-version',
  'prompt_template': 'chat-template-name',
  'context_length': 8192,
  'temperature': 0.0,
  'top_p': 1.0,
  'seed': 42,
  'gpu': '24GB-class GPU',
  'power_limit_w': 300
}

The configuration should be treated as part of the experiment. If you change the runtime version or the prompt template, you are running a different experiment. That is fine, but it should be a deliberate variable, not an accidental one.

Why model identity is not just the model name

Local model files are often distributed under names that hide important details. A repository may contain several quantization files, and a chat UI may apply a default system prompt that is not part of the model. If you compare two models using different default prompts, you are measuring the prompt as much as the model. If you compare a local model with a hosted API, the hosted service may apply its own system prompt, safety filters, tokenization, and routing. Hermanity’s guide notes that hosted and subscription models run through their real access paths in its benchmark, which is a reminder that the access path is part of the system under test.

Choose tasks that resemble real work

A task set is a small, representative sample of the work the model must do. It should include normal cases, edge cases, and failure cases. For a first experiment, keep the task set narrow and well-defined. For a durable benchmark, expand it as the decision becomes more important.

Hermanity’s guide describes a combination of custom operational tasks with recognized industry suites. That hybrid approach is useful because industry suites provide comparability with other work, while custom tasks reflect your actual workflow. A local model may pass a general reasoning benchmark but fail to produce a valid WordPress post excerpt or a JSON object for an automation pipeline. The custom tasks catch that.

Common task categories for local model comparisons include:

• Extraction and classification: pull fields from text, label sentiment, detect spam, or route a request. These tasks often have clear correct answers and low tolerance for format errors.
• Summarization: condense a document while preserving facts, names, numbers, and tone. Quality is often judged with a rubric rather than exact match.
• Code generation and debugging: write a function, fix an error, or explain a stack trace. Correctness may be checked by tests or human review.
• Structured JSON: return machine-readable output for an automation workflow. Schema validity and key completeness are primary metrics.
• Long-context question answering: answer questions using a long document. This tests both retrieval and memory behavior.
• Creative and editorial work: draft, rewrite, or adjust tone. Subjective quality requires human review or a carefully designed rubric.
• Tool or agent behavior: choose a tool, call an API, or follow a multi-step instruction. Failure modes include wrong arguments, loops, and unsafe actions.

For each task, define the input, the expected output, the scoring method, and what counts as a failure. A failure might be an invalid JSON object, a missing required field, a factual contradiction, a refusal, a timeout, or an out-of-memory error. Treating failures as a separate metric is important because a model that is fast but fails often is not reliable.

Use a prompt template that matches how the model will be used in production. If the production system uses a system prompt, include it. If it expects JSON, ask for JSON in the same way. If it uses a specific chat format, apply that format consistently. Changing the prompt between models may be necessary if each model requires a different template, but then the prompt template becomes part of the comparison. Record it.

Designing a small but meaningful task set

A pilot task set can be small, but it should be chosen carefully. Start with examples that cover the range of inputs the model will see: short and long documents, clean and noisy text, simple and ambiguous instructions, and cases where a wrong answer would be costly. Include at least a few examples that are known to be difficult. If all examples are easy, every model will look good and the comparison will not support a decision.

The right number of examples depends on the effect size you need to detect and the variability of the task. A small set can reveal large differences; a larger set is needed when models are close. For a first pass, a compact set of representative cases is often enough to eliminate clearly unsuitable models. Expand the set when the decision is close or the cost of a wrong choice is high.

Build a controlled test harness

A test harness is the machinery that runs the tasks and records the results. It does not need to be elaborate. A script, a configuration file, a task file, and a results directory are enough to start. The key is that the harness enforces the same procedure for every candidate.

A practical first harness can follow this sequence:

1. Create a directory structure. Separate configurations, task sets, raw outputs, scores, and logs. This makes it possible to rerun or audit a comparison.
2. Write a configuration file. Include every pinned field from the model identity and environment checklist. Use one configuration per model or per experimental condition.
3. Prepare the task set as structured data. A JSON Lines file is a common choice: one JSON object per line, with an input and any expected output or metadata. This is easy to parse and easy to inspect.
4. Write a runner that sends the same prompt template and sampling parameters. The runner should load the task, format the prompt, call the runtime, and capture the raw response.
5. Warm up the model. The first request after loading can be slower because of model loading, kernel compilation, or cache initialization. Run a warm-up request and discard it from timing results.
6. Run each model and task combination multiple times. Multiple runs expose variance. If the model supports a seed, set it; if not, record that it is unavailable.
7. Log raw outputs, timings, token counts, memory, and errors. Do not log only the score. The raw output is needed to understand why a model failed.
8. Score automatically where possible and manually where necessary. Combine exact checks, schema validation, and human review for subjective tasks.

Keep the harness simple enough that you can explain it. A complex harness with hidden defaults is harder to reproduce than a small script with an explicit configuration.

What to log for every run

At minimum, log the experiment name, timestamp, model identifier, configuration hash or version, task identifier, prompt template name, sampling parameters, raw input, raw output, time to first token, total generation time, token counts, peak memory if available, and any error or exception. If the runtime exposes a served model identifier, log that too. If you use a scoring script, log the scorer version and the score.

This log becomes the evidence base. When a result looks surprising, the log lets you check whether the model actually changed, whether the prompt was different, or whether a background process affected the run.

Measure quality, speed, memory, and failure rates

A fair comparison uses several dimensions. Quality alone can hide unacceptable latency. Speed alone can hide invalid output. Memory alone can hide poor task performance. Failure rates reveal reliability problems that averages can mask.

Quality scoring depends on the task. For extraction and classification, exact match or field-level accuracy may work. For JSON, validate against a schema and count missing or invalid fields. For summarization, use a rubric that checks factual consistency, coverage, length, and tone. For code, run tests when possible and use human review for style and maintainability.

LLM-as-judge can help scale subjective scoring, but it introduces its own biases. If you use a judge model, record its identity, prompt, and sampling settings. Validate the judge against a small set of human-scored examples. Do not treat a judge score as ground truth.

Latency should be reported as a distribution, not only an average. A model with a good average but a long tail may be frustrating in interactive use. Time to first token matters for chat and streaming interfaces; total generation time matters for batch jobs. Measure both when they are relevant.

Memory estimates must include more than the model weights. Context length increases the KV cache, and the runtime adds overhead. A model that fits at a short context may run out of memory at a longer context. If you cannot measure peak memory directly, record the context length and note that memory is estimated. Do not present weight-file size as total memory use.

Run fair comparisons

Fairness comes from controlling variables. If you are comparing quantizations, keep the runtime, prompt template, sampling parameters, context length, and hardware the same. If you are comparing runtimes, keep the model and quantization the same. If you are comparing hardware, keep the software stack the same. Change one major variable at a time, or explicitly design a factorial experiment if you need to study interactions.

Run the candidates in an interleaved order when possible. Running all of model A and then all of model B can confound results with thermal throttling, background updates, or cache state. Alternating runs or randomizing order helps reduce that risk. If randomization is not practical, at least run a warm-up and monitor system conditions.

Repeat runs and report the spread. A single run is not enough to compare models, especially when sampling is enabled. If temperature is greater than zero, output variance is expected. Even at temperature zero, runtime nondeterminism can produce small differences. Multiple runs show whether a difference is stable or noise.

Be careful when comparing a local model with a hosted API. The hosted service may use a different prompt template, system prompt, tokenizer, safety layer, or model version. Network latency and rate limits affect the user experience. Hermanity’s guide notes that hosted and subscription models run through their real access paths, which is the right approach if the decision is about routing between local and hosted options. Align the task and prompt as closely as possible, and measure the end-to-end path that users will actually experience.

Analyze and report honestly

Analysis starts with the decision criteria you wrote before the test. Did the model meet the quality threshold? Did it stay within the memory ceiling? Did it meet the latency budget? Did it fail in ways that require human intervention? A model that wins on average but fails unpredictably may be worse for a production workflow than a slightly slower model with consistent output.

Report all runs, not just the best one. If a model produced a perfect answer once and failed twice, the failure rate matters. If a model was faster in one run but slower in the next, the distribution matters. Use ranges, percentiles, or simple summaries that show variation. Avoid presenting a single cherry-picked output as representative.

Separate observations from hypotheses. An observation is what the log shows: model A produced invalid JSON on 3 of 20 cases. A hypothesis is an explanation: model A may need a stricter prompt or grammar-constrained decoding. Both are useful, but they should not be confused. The next experiment can test the hypothesis.

Include the environment and versions in the report. A result without the runtime version, quantization, context length, and hardware is difficult to reproduce. If the result depends on a specific prompt template, include it. If the task set is private, describe its composition and provide aggregate results without exposing sensitive data.

State the limitations. The task set is a sample, not the whole workload. The hardware is one configuration, not all configurations. The quantization is one format, not all formats. The judge model may have biases. The results are evidence for a decision, not a universal ranking. Hermanity’s guide makes the same point: the goal is not to crown one universal winner but to build evidence for routing decisions.

Worked example: comparing two quantized models for a WordPress editorial workflow

The following example is hypothetical and illustrative. It shows how the concepts fit together; it is not a report of a tested environment.

Suppose a small editorial team wants to use a local model to help with WordPress drafts. The workflow has three tasks: summarize a draft into two sentences, generate a meta description, and classify comments as spam or not spam. The team has one 24 GB GPU and two candidate local checkpoints. Both checkpoints are available in multiple quantizations. The decision is which model and quantization to use for a private editorial assistant that runs on the team’s own hardware.

The team writes the decision first: the chosen model must produce factually consistent summaries, valid JSON for the automation step, and responses within a latency budget that keeps the editor’s workflow responsive. It must fit in 24 GB of VRAM at the chosen context length. If both models meet the criteria, the team will prefer the one with lower memory use or a clearer license for commercial use.

Next, the team pins the environment. It records the exact checkpoint, quantization, runtime and version, prompt template, context length, sampling parameters, GPU model, memory, and power limit. It creates a configuration file for each candidate and stores it with the task set.

The task set contains representative WordPress drafts, including short posts, long posts, posts with lists, and posts with names and numbers that must not be changed. It also contains comments with obvious spam, borderline spam, and legitimate criticism. Each task has a defined output format. The summarization task asks for a JSON object with a summary and a meta description. The classification task asks for a label and a confidence value.

The harness runs each model on each task three times. It logs the raw output, JSON validity, summary length, time to first token, total generation time, and peak memory. A human editor reviews the summaries for factual consistency and tone. The classification labels are checked against a small human-labeled set.

After the runs, the team compares the results. If model A produces better summaries but fails JSON validation more often, the team may test grammar-constrained decoding or a stricter prompt. If model B is faster but uses more memory at long context, the team may test a lower context length or a different quantization. If both models meet the criteria, the team chooses based on memory, license, or maintainability rather than declaring a universal winner.

In a WordPress automation context, the test should also cover the integration path. If the model output feeds a plugin or a queue, validate the JSON schema, handle retries, prevent duplicate posts, and log failures for review. The benchmark should include the same input and output boundaries that the automation will use. A model that works in a chat window but fails to produce valid JSON for the automation is not a good fit for that workflow.

Troubleshooting and common mistakes

Most comparison problems have a small set of causes. The following symptoms, diagnostics, and actions cover the common cases. Treat causes as hypotheses until the logs confirm them.

Symptom: results differ between runs

Plausible cause: sampling parameters are not fixed, the runtime is nondeterministic, or the model or runtime changed between runs.

Diagnostic check: compare the configuration files and logs. Check temperature, top-p, top-k, seed, runtime version, and model digest.

Action: set temperature to zero if the task needs deterministic output, fix the seed if supported, and use the same runtime version for all candidates. Run multiple trials and report the variance that remains.

Verification: rerun the same configuration and confirm that the results are comparable, even if not byte-identical.

Symptom: the model is slower after the first prompt

Plausible cause: thermal throttling, background processes, or a cold start that was not excluded from timing.

Diagnostic check: monitor GPU clocks, temperature, and system load during the run. Check whether the first run was slower than later runs.

Action: add a warm-up request, close unnecessary background applications, and interleave candidate runs to distribute thermal effects.

Verification: compare p50 and p95 latency across runs. If the slowdown persists, investigate the runtime or hardware configuration.

Symptom: out-of-memory errors at long context

Plausible cause: the KV cache grows with context length, and the total memory includes weights, cache, and runtime overhead.

Diagnostic check: record the context length and peak memory. Test a shorter context to see whether the error disappears.

Action: reduce context length, use a more memory-efficient quantization, adjust KV-cache settings, or offload layers if the runtime supports it.

Verification: rerun the same task at the new setting and confirm that memory stays within the hardware limit.

Symptom: quality drops after quantization

Plausible cause: the task is sensitive to quantization, or the quantization format interacts poorly with the runtime.

Diagnostic check: compare the same checkpoint at different quantization levels using the same runtime, prompt, and sampling settings.

Action: test multiple quantization levels on the actual task. Do not assume that a larger quantized model always beats a smaller higher-precision model; the result depends on the task and memory constraints.

Verification: check whether the quality difference is consistent across runs and task examples.

Symptom: JSON output is often invalid

Plausible cause: the prompt does not specify the schema clearly, the model is not suited to structured output, or sampling introduces formatting errors.

Diagnostic check: inspect the invalid outputs. Look for missing fields, extra text, markdown fences, or truncated responses.

Action: use a schema, provide a clear example, use grammar-constrained or structured decoding if the runtime supports it, and add a validation-and-retry step.

Verification: measure the schema-valid rate before and after the change.

Symptom: a leaderboard favorite fails the real task

Plausible cause: the leaderboard task does not match your workflow, the prompt format is different, or the model was evaluated under different quantization and runtime conditions.

Diagnostic check: compare the leaderboard’s task format with your task set. Check whether your prompt template matches the model’s expected chat format.

Action: treat leaderboard scores as a screening signal, not a final decision. Run your own task set with your own environment.

Verification: confirm that the model’s performance on your task set supports the decision you need to make.

Symptom: local versus hosted comparison feels unfair

Plausible cause: the hosted service uses a different system prompt, model version, safety layer, or tokenizer, and network latency is included in the user experience.

Diagnostic check: document the access path for each option. Compare prompts, model identifiers, and measured end-to-end latency.

Action: align the task and prompt as closely as possible, measure the real access path, and include cost, privacy, and reliability in the decision.

Verification: rerun the comparison with the same task set and report the remaining differences as part of the access path.

Responsible operation and limits

Reproducible testing improves decisions, but it does not remove responsibility. Local inference can keep data on your hardware, which is a privacy advantage, but logs and outputs may still contain sensitive information. Redact or encrypt logs if they include private documents. If you use a hosted model for comparison, review its data handling terms.

Licensing is another practical limit. Open weights, open source, and permission for commercial use are not the same thing. Check the applicable license for each model and any restrictions on redistribution or commercial deployment. A model that performs well but has an unsuitable license is not a viable choice for a commercial workflow.

Security matters when downloading model files and runtimes from third parties. Prefer official repositories and verify checksums or digests where possible. A model file is executable in the sense that it is interpreted by a runtime; supply-chain risks are real.

Finally, remember that a benchmark is a sample. It can support a decision, but it cannot guarantee future behavior. Model updates, runtime updates, prompt changes, and workload changes can alter results. Revisit the comparison when the environment changes or when the cost of a wrong choice increases.

A practical first experiment

If you are new to reproducible local model comparison, start with one decision and two candidates. The following sequence is a realistic first path.

1. Write the decision. Example: choose a local model for private draft summarization on a 24 GB GPU.
2. Choose two candidates. Pick one smaller model and one larger model, or two quantizations of the same checkpoint. Keep the comparison focused.
3. Pin the environment. Record the runtime, version, prompt template, context length, sampling parameters, GPU, memory, and power limit.
4. Create a small task set. Use representative drafts and define the expected output format. Include a few difficult cases.
5. Build a simple harness. A script that reads the task set, calls the runtime, and writes raw outputs and timings is enough.
6. Run a warm-up and then three trials per model. Interleave the models if possible.
7. Score quality, latency, memory, and failures. Use automatic checks for format and human review for factual consistency and tone.
8. Report the results with the configuration. Include the raw outputs or a summary, the environment, and the limitations.

Success checks for this first experiment are simple. Can you rerun it and get comparable results? Can you explain why one model performed better on a specific task? Do you have logs that show what happened? If the answer is yes, you have a reproducible test. If the answer is no, the next step is to fix the missing variable, not to collect more models.

Where to go next

After the first experiment, expand in the direction that matters. If memory is the constraint, study KV-cache behavior and quantization tradeoffs. If quality is the constraint, improve the task set and rubric. If reliability is the constraint, add failure-rate metrics and retry logic. If the goal is WordPress automation, test the full integration path: input preparation, model call, JSON validation, queue handling, duplicate prevention, editorial review, and logging.

Useful supporting topics for this pillar include how to measure VRAM and KV cache for local LLMs, how quantization levels affect local inference, how to build a local LLM test harness with structured logs, how to design human review rubrics for AI summaries, and how to use local models for WordPress editorial automation. Each of those topics can deepen one part of the comparison without repeating the whole guide.

If you want community feedback, share your goal, hardware, runtime, model identifiers, task set description, attempted steps, and observed results. A specific report makes it easier for others to spot missing variables or suggest a fair next test. The point is not to collect opinions about which model feels best; it is to build evidence you can inspect, repeat, and use.