How to Evaluate Hallucinations in a Local Model with Your Own Test Set

A hallucination evaluation for a local model does not require a benchmark suite or a research team. It requires three things: a small file of questions whose correct answers you have already verified, a way to send those questions to the model with no tools and no retrieval, and a grading rule that turns each answer into one of three verdicts — correct, hallucinated, or hedged. The share of answers marked hallucinated is your model’s hallucination rate on that domain.

This guide is for practitioners running models locally through Ollama, vLLM, LM Studio, or llama.cpp who want a number that reflects their own use case rather than a leaderboard. The loop below follows the evaluation method documented by Groundtruth Data’s runner, and the alternative grading path uses DeepEval, an open-source LLM evaluation framework. No measurements are reported here; everything below is a protocol you run yourself, on your hardware, against your questions.

What a hallucination test set actually measures

Three quantities get confused with one another, and separating them is the first useful step.

• Accuracy is the share of questions answered correctly. It says nothing about what the model does when it does not know.
• Hallucination rate is the share of questions answered confidently and wrongly. This is the number most teams care about, because a confident wrong answer is harder to catch downstream than a refusal.
• Hedge rate is the share of questions where the model expresses uncertainty or declines. A high hedge rate lowers the hallucination rate but can also make the model useless for the task.

Groundtruth’s loop counts exactly these three verdicts and treats the hallucinated percentage as the model’s hallucination rate on that domain. That framing matters: the number is domain-specific. A model can be reliable on general trivia and unreliable on your internal configuration details, and only your own test set will show the difference.

Prerequisites

Before writing questions, assemble the following. Each item exists to make the run reproducible.

1. A local model server with an OpenAI-compatible endpoint. Ollama, vLLM, LM Studio, and llama.cpp all expose one. Groundtruth documents the runner as working with any model reachable through an OpenAI-compatible base URL, alongside first-class paths for the Anthropic and OpenAI APIs.
2. A runner or a short script. Either a JSONL runner or a Python harness such as DeepEval.
3. A grading rule. Exact match for numbers and dates; a judge-model call for prose.
4. A configuration record. Model name and tag, quantization, context length, runtime version, temperature, top-p, and seed. Without this, two runs are not comparable.
5. A place to store results with one row per question and a verdict column.

Building your own test set

Where the questions come from

The value of a custom set is that it asks what you actually ask. Good sources are your own documentation, product manuals, support tickets, configuration notes, and the questions colleagues send you in chat. Write each question as a fair, open question that a competent person could answer from a primary source — not a trick question, and not a question whose answer is ambiguous.

The anatomy of a row

Groundtruth ships each row as a complete, machine-runnable test case: a fair open question, the verified answer re-derived from the primary source, the source URL, grading context, and a frontier model’s graded attempt as a baseline. Mirroring that structure is the cheapest way to keep your set honest, because the source URL forces you to check the answer rather than recall it.

A template row in JSONL — one object per line — looks like this:

{"question": "What context length is configured in our deployment?", "expected_answer": "8192", "source_url": "https://internal.example/config-notes", "grading": "exact"}

The values above are placeholders. The important part is that expected_answer was copied from a document you can point at, and grading tells the harness whether to compare exactly or send the answer to a judge.

How many rows

No source prescribes a fixed count, and a number chosen for statistical comfort is usually larger than a first pass needs. A practical approach is to start with a set small enough to write in an afternoon, stratified across the topics you care about, and stable enough that you can re-run it after every change. Keep a portion of the rows out of any prompt-tuning work so the score remains a measurement rather than a target.

Running the model cold

"Cold" means no tools, no retrieval, and no system prompt that smuggles in the answer. Groundtruth states the reason directly: the answer should reveal what the model actually knows instead of what it can look up. If you run the same questions later with retrieval enabled, the gap between the two scores is what retrieval buys you.

Option A: a JSONL runner

1. Save your rows as my-domain.eval.jsonl, one JSON object per line.
2. Point the runner at your local server. Groundtruth documents --provider, --base-url, and --model for OpenAI-compatible endpoints:

node eval-runner.mjs my-domain.eval.jsonl 
  --provider openai 
  --base-url http://localhost:11434/v1 
  --model llama3.1:8b

The port, path, and model tag depend on your runtime; check its OpenAI-compatibility documentation rather than copying the example blindly. For hosted APIs the same runner is documented with ANTHROPIC_API_KEY=... node eval-runner.mjs file.eval.jsonl or --provider openai --model gpt-4o.

3. Record the configuration you used next to the results file. A hallucination rate without a configuration record is not evidence.

Option B: DeepEval

DeepEval is an open-source framework described as similar to Pytest but specialized for unit testing LLM outputs. It ships ready-to-use metrics including hallucination, faithfulness, answer relevancy, G-Eval, RAGAS, contextual recall and precision, toxicity, and bias, and its metrics run locally on your machine. It can evaluate a dataset in bulk in under 20 lines of Python, in parallel, either through a Pytest-like CLI or an evaluate() function.

The shape of a run is roughly this; import paths and constructor arguments change between releases, so check the documentation for your installed version:

from deepeval import evaluate
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What context length is configured in our deployment?",
    actual_output=model_answer,
    context=["8192"],
)

metric = HallucinationMetric(threshold=0.5)
evaluate([test_case], [metric])

There is an important conceptual difference between the two paths. DeepEval’s hallucination metric compares the model’s output against supplied context, which makes it a faithfulness-style check: it asks whether the answer is grounded in the context you provided. The JSONL loop instead compares against a verified expected_answer, which measures whether the model knows the fact at all. Both are useful, and they answer different questions.

Grading: exact match, judge, and three verdicts

Groundtruth’s grading rule is deliberately cheap: exact match for numbers and dates, or one judge-model call for prose. Then count the verdicts.

These categories are illustrative definitions, not measured results. Practical grading notes: normalize numbers and dates before comparing; keep the judge prompt fixed across runs; log the judge’s rationale so you can audit borderline calls; and write down what counts as hedged before you start, because that boundary is easy to move unconsciously when the results disappoint.

Reading the results

Once every row has a verdict, four questions become answerable.

• What is the hallucination rate on this domain? Report it with the configuration and the question set, not on its own.
• Did something regress? Groundtruth lists catching regressions before shipping a fine-tune as a primary use of the loop. Re-run the same file after a quantization change, a runtime upgrade, or a prompt change.
• What does retrieval buy? Run the same questions with retrieval on; the gap between the two scores is the retrieval benefit.
• Is the model calibrated? A model that hedges when unsure is safer than one that confidently invents. Track the hedge rate next to the hallucination rate — a model that hedges everything scores well and helps nobody.

Limitations and pitfalls

• Contamination. If your questions or answers appear in the model’s training data, the score is inflated. Prefer questions drawn from material published after the model’s training cutoff, or from internal documents.
• Version and quantization drift. The same model name at a different quantization is a different artifact. Test the artifact you deploy.
• Sampling settings. Temperature and seed affect repeatability. Fix them, or run the set more than once and report the variation.
• Judge bias and cost. A judge from the same model family as the model under test may be lenient. Spot-check a sample by hand, and remember that judge calls are the main cost of the prose path.
• Small samples. A handful of rows produces a noisy rate. Stratify by topic and keep the set stable so changes over time mean something.
• Stale answers. Dates, prices, and version numbers change. Re-derive expected answers from primary sources periodically.
• Tuning on the test set. If you edit prompts until the set passes, it stops being a test. Keep a held-out portion.

Where to start

Write twenty questions from your own documentation, verify each answer against a source you can link, and run them cold against the model you actually deploy. Grade them, record the configuration, and write down the hallucination rate and the hedge rate. Then run the same file with retrieval enabled and compare. That two-run comparison is more informative than any general benchmark, because it measures the system you are shipping rather than a model in the abstract.

If you run this on your own hardware, the community is the right place to compare notes: share your goal, your environment (runtime, model tag, quantization, context length), the result you got, and any error you hit along the way. Reproducible details from other people’s setups are what turn a personal score into a useful reference.