A repeatable AI test harness is the plumbing that turns a set of examples and some graders into runs you can re-execute and compare. It is deliberately unglamorous: mostly bookkeeping, versioning, and result storage. But getting that bookkeeping wrong is the most common reason evaluation numbers become uninterpretable, so the harness matters as much as the prompts it tests.
This guide is for developers and prompt engineers who already run ad hoc spot checks in a chat window and want to turn them into a dependable part of the release process. It covers the four parts every harness needs, why they must be separable, what to pin for reproducibility, and the tradeoffs that make harnesses harder than they first appear.
What a test harness actually does
A prompt testing harness is a lightweight evaluation layer that sits between your prompts and your release process. Given a set of known inputs, it runs your prompt or workflow, captures the model output, and checks whether the result still meets your expectations. That sounds basic, but it solves a real problem: prompts are not static assets. You revise instructions, add examples, swap models, change system messages, shorten context to save tokens, or add retrieval to improve grounding. Each change can improve one case while quietly breaking another. A harness helps you see those tradeoffs before users do.
This approach matches a core principle of prompt engineering for developers: treat prompts more like functions than one-off conversations. A harness formalizes that loop by making prompt changes testable the same way code changes are.
The four parts of a harness
Whatever you build it with, a harness has four parts:
- A dataset: versioned examples, with expected outputs or grading criteria.
- A runner: executes your system against each example, with concurrency, retries, and caching.
- Scorers: one or more graders per example, from code checks to judges.
- A result store: every run, with enough metadata to reproduce it and to compare it with another.
The Inspect framework names the same four parts slightly differently, and its naming is worth borrowing even if you never use the tool: a dataset holds your examples; a Solver is the thing being tested—your prompt, chain, or agent; a Scorer is the grader that marks the solver’s answer; and a Task is the bundle that says “run this dataset through this solver and mark it with this scorer.”
Why separability is the property that pays off
The important property is that the four parts are separable. That pays off in two everyday situations. You want to try a new model: swap the solver, and the dataset and scorers stay untouched. You think of a new check: add a scorer and re-mark yesterday’s saved outputs, with no need to pay for generating them again. A harness where these are tangled together makes both of those a rewrite instead of a one-line change.
The fourth part—the result store—is the one teams skip and the one that determines whether the harness is worth having. Saving raw outputs alongside the metadata that produced them is what lets you re-score without regenerating, and what lets you compare two runs honestly.
Pin everything that can change an answer
An eval score is only meaningful relative to the conditions that produced it. If you change the model, the sampling temperature, the prompt template, the retrieval index, or the grader between runs, the score difference no longer tells you what changed. Pin the conditions you want to hold constant, and record the ones you vary.
A practical minimum to record per run:
- Prompt template and system message, with a version identifier.
- Model name and version or snapshot identifier.
- Sampling settings (temperature, top-p, max tokens, seed if supported).
- Context length or truncation policy.
- Retrieval configuration and index version, if retrieval is used.
- Dataset version and scorer version.
- Runtime and harness code version.
Without this metadata, a score is a number without units. With it, a regression is a diff you can act on.
What makes a dataset usable
A dataset earns its keep by covering the cases you actually care about. Representative inputs drawn from real usage, edge cases, and known past failures are more valuable than a large set of easy examples. Each example should carry either an expected output or explicit grading criteria, because a scorer cannot mark what the dataset never defined.
Version the dataset itself. When you add examples, tag the change so a score jump can be traced to a dataset edit rather than a prompt edit. If your product has sensitive or regulated outputs, keep a small human-reviewed subset to check that automated scorers still agree with human judgment.
An illustrative harness layout
The following is a proposed file layout, not a tested implementation. It shows how the four parts stay separate:
harness/
datasets/
support-qa.v3.jsonl # versioned examples
solvers/
prompt_v4.py # the thing being tested
scorers/
exact_match.py
llm_judge.py
runs/
run-0007_prompt_v4_model-x/
config.json # pinned conditions
outputs.jsonl # raw model outputs
scores.jsonl # grader results
A single dataset example might look like this:
{
"id": "qa-014",
"input": "How do I reset my password?",
"expected": "Use the 'Forgot password' link on the sign-in page.",
"criteria": ["mentions the forgot-password link", "no invented phone number"]
}
Because outputs are stored separately from scores, you can add a new scorer later and re-mark outputs.jsonl without calling the model again.
Choosing scorers: code checks versus judges
Scorers range from deterministic code checks to model-based judges. Code checks—exact match, regex, JSON schema validation, required-substring checks—are cheap, fast, and reproducible, and they should carry as much of the load as possible. Judges are useful when the criterion is fuzzy, such as tone, helpfulness, or factual grounding, but they introduce their own variance and cost.
A few practical habits help:
- Prefer a code check whenever the criterion can be expressed as one.
- When using a judge, pin the judge model and prompt, and treat the judge version as part of the run conditions.
- Run multiple scorers per example rather than collapsing everything into one pass/fail. Different scorers catch different regressions.
- Keep a small set of hand-labeled examples to sanity-check the judge against human judgment.
Detecting regressions and comparing runs
Once outputs and scores are stored per run, comparison becomes a query rather than a re-run. A useful workflow is to diff the scores of the current run against a baseline run, then inspect the examples whose scores changed. Because raw outputs are saved, you can read the before-and-after text directly instead of guessing what changed.
Report per-example results, not just an aggregate pass rate. An aggregate can hide a swap where three cases improved and three broke. Per-example diffs make the tradeoff visible, which is exactly the decision a harness exists to support.
Wiring the harness into your workflow
The harness becomes valuable when it runs without being remembered. A common pattern is a fast suite that runs on every prompt or model change, and a larger suite that runs before a release. If you maintain content or automation pipelines—for example, a WordPress site that uses AI-generated drafts—the same harness can gate the pipeline: run the suite, and only publish or schedule when the scores meet your threshold.
Keep the harness runnable from a single command so it can be triggered by a script, a scheduled job, or a CI step. Store each run under a stable identifier so results remain comparable over time.
Limitations and tradeoffs
A harness is not a guarantee of quality. A dataset that does not represent real traffic will pass changes that break users. A judge with a fixed prompt can drift in behavior when its underlying model changes. Caching can speed up runs but can also mask nondeterminism if it is not keyed on every input that matters.
There is also a cost tradeoff. Running every example against every model variant gets expensive quickly; most teams run a small, fast suite on every change and a larger suite less often. The right balance depends on how costly a regression is in your product.
Finally, a harness measures what you chose to measure. If your dataset lacks adversarial or edge cases, the harness will report green while the edge cases remain untested. Treat the dataset as a living artifact with its own version history.
Where to start
You do not need a framework to begin. Start with a versioned JSONL file of representative inputs, a script that runs your prompt against each one, one or two code-based scorers, and a directory that stores raw outputs plus a config file. That minimal harness already gives you the two things that matter most: the ability to re-run a comparison, and the ability to re-score old outputs without paying for generation again.
Once that loop is stable, add a judge for the criteria code cannot express, split your suite into fast and full tiers, and make the harness run automatically on prompt or model changes. If you are running local models, pin the quantization and runtime version too, since those change outputs as surely as a model swap does.
If you have built a harness for a specific stack, sharing your dataset format, scorer choices, and the failure modes you hit is often more useful to the community than a polished framework. Consider posting your setup, environment, and any error messages in the Ai-Forum community so others can reproduce and compare.
FAQ
How is a test harness different from a unit test?
A unit test checks deterministic code behavior, while a harness runs a dataset of examples through a model-based system and scores outputs that may vary. The harness also stores raw outputs and run metadata so results can be re-scored and compared across model or prompt changes.
Can I re-score old outputs without regenerating them?
Yes, if the result store keeps raw model outputs separately from scores. Adding a new scorer then becomes a re-marking pass over saved outputs rather than a new generation run.
Do I need a framework like Inspect to build a harness?
No. A versioned dataset file, a runner script, one or two code-based scorers, and a directory of run records form a minimal working harness. Frameworks can add convenience, but the four-part structure is what makes the harness useful.
What should I version in a harness?
At minimum, version the dataset, prompt template, model identifier, sampling settings, retrieval configuration if used, scorer versions, and runtime. Any of these can change an answer, so an unversioned change makes score differences ambiguous.