Measuring a hallucination rate in production means turning a vague worry — the model sometimes makes things up — into a tracked number: the share of sampled outputs that assert something the available evidence does not support. The method that observability tooling converges on is a two-part loop. A deterministic pre-screen catches cheap, obvious risk signals for free, and a reference-free LLM-as-a-judge evaluator scores a sample of live traffic against the input and context recorded on the same observation, so no ground truth is required (Langfuse).
This guide is for practitioners who already run an AI workflow — a RAG assistant, a drafting pipeline, an agent that calls tools — and want a rate they can put on a dashboard instead of an anecdote. It covers what to count, how to assemble the pipeline, how to validate the judge, and where the resulting number can mislead you.
What a hallucination rate actually measures
Langfuse defines a hallucination as a claim in the output that the available evidence does not support. Two distinctions determine how you detect one, and both change what your denominator should be.
Intrinsic versus extrinsic. An intrinsic hallucination contradicts the source material: the context says the limit is 100 requests per minute and the answer says 1,000. An extrinsic hallucination adds material the source does not contain: the answer cites a paper that was never retrieved, or invents a configuration flag. Extrinsic claims may even be true, but an application that promises grounded answers cannot verify them, so detection treats unsupported and false the same way.
Grounded-context versus open-domain. The second axis is what evidence exists to check the claim against. A grounded-context workflow supplies retrieved documents, database rows, or tool output; an open-domain workflow does not. Grounded workflows can be scored reference-free, because the retrieved context is itself the evidence. Open-domain workflows need a reference answer, a rubric, or human review.
The measurement consequence is simple: state whether you are counting flagged responses or flagged claims, and store each scored output next to the input and context that were available when it was generated. Without that evidence pointer, a reference-free judge has nothing to check.
The two layers of a measurement system
Fiddler’s cookbook describes hallucination detection as a two-layer pipeline: pre-deployment evaluation with the Evals SDK, testing against known scenarios and validating with golden labels, and production monitoring through LLM Observability enrichments and Evaluator Rules that watch live traffic continuously. Langfuse describes the same split from the other direction: offline detection runs experiments against a dataset with reference answers before each release, where a judge can verify claims against known-correct outputs, while production detection runs a reference-free judge on a sample of live traffic.
| Layer | Evidence it uses | Question it answers |
|---|---|---|
| Pre-deployment (offline) | Dataset with reference answers or golden labels | Did this prompt, model, or retrieval change make grounding worse? |
| Production (online) | Live traffic, reference-free judge against recorded context | What is the current rate on real inputs? |
Both layers earn their place. Offline catches regressions before release; production catches drift and input patterns your dataset never covered.
Building the production pipeline
1. Instrument the workflow so evidence travels with the output
Log the input, the retrieved context or tool output, and the final answer on the same observation or trace. This is the prerequisite for everything else. Langfuse’s production pattern is explicit: the judge is reference-free and checks the output against the input and context recorded on the same observation.
2. Pre-screen deterministically
Deterministic code pre-screens flag risk signals for free and route only flagged traces to the paid judge. Typical checks: a citation key that does not exist in the retrieved set, a number in the answer that appears nowhere in the context, a URL that was not among the sources. These are heuristics, not verdicts — they produce candidates, and their false-positive rate is a number worth watching.
3. Sample live traffic and run the judge
Judging every trace is expensive, so production detection scores a sample. Use two streams: all pre-screen-flagged traces, which are high yield, and a random sample of unflagged traces, which estimates what the pre-screen misses. Judging only flagged traces gives you a rate over a biased subset that cannot be extrapolated to total traffic.
4. Score binary and record the failing claims
Langfuse recommends scoring verdicts as binary hallucinated or clean, with the failing claims listed in the score comment. Binary verdicts aggregate cleanly into a rate; the comment preserves the detail you need to debug a spike.
5. Compute the rate against the sampling design
The rate is the estimated number of hallucinated outputs divided by total outputs — not the share of judged outputs that were flagged, unless you judged everything.
An illustrative calculation
The arithmetic below is a worked example of the estimator, not a measured result from any deployment.
Suppose a workflow produces 1,000 outputs a day. A deterministic pre-screen flags 40. You run the judge on all 40 and on a random sample of 100 of the remaining 960. The judge marks 30 of the 40 flagged outputs as hallucinated, and 5 of the 100 unflagged outputs.
Estimated hallucinated outputs = 30 + (5 ÷ 100 × 960) = 30 + 48 = 78, or about 7.8% of 1,000 outputs.
The estimate rests on one assumption worth stating out loud: the random sample represents the 960 outputs the pre-screen did not flag. It also carries real uncertainty — 5 out of 100 is a small sample, and the interval around it is wide. Report the sample size alongside the rate.
Validating the judge before you trust the number
The judge is itself a model and can be wrong in both directions. Two practices keep the number honest.
- Hand-label a sample and compare. Take 30 to 50 outputs, label them yourself or with a domain expert, and record agreement. Track false positives (a clean output marked hallucinated) and false negatives separately, because they bias the rate in opposite directions.
- Separate failure modes with the RAG triad. Fiddler’s cookbook points to AnswerRelevance, ContextRelevance, and RAGFaithfulness as a health-metrics triad that distinguishes hallucinations from other failure modes. An answer that is relevant but not faithful to its context points at generation; a faithful answer built on irrelevant context points at retrieval.
The Fiddler cookbook installs its evaluation packages with pip install fiddler-evals fiddler-client pandas and imports the evaluators directly:
from fiddler_evals import init, evaluate, Project, Application, Dataset
from fiddler_evals.evaluators import (
AnswerRelevance,
ContextRelevance,
RAGFaithfulness,
)
The cookbook’s setup also requires an LLM credential configured in the account’s LLM Gateway settings, because the evaluators call a model to do the scoring.
Limitations and tradeoffs
- Sampling error. A rate from 100 sampled outputs is an estimate, not a census. The sources do not prescribe a sample size; larger samples narrow the interval, so pick a fixed weekly sample you can afford and widen it as volume grows.
- Judge cost and latency. Every judged trace is an extra model call. The deterministic pre-screen exists precisely to keep that bill down.
- Unsupported but true claims. Because detection treats unsupported and false the same way, your rate will include some claims a human would not call a lie. Say so when you publish the number.
- Context quality. If retrieval returns the wrong documents, an answer can be faithful to bad context and still fail a grounded check. Pair the hallucination rate with context metrics rather than reading it alone.
- Drift after changes. A new generator model, prompt, or index invalidates the judge’s calibration. Re-run the agreement check after each material change.
- Binary verdicts lose severity. An invented API method and a minor unverifiable aside score identically. The failing-claims comment is what preserves severity for triage.
Running the judge locally, and measuring inside WordPress automation
A self-hosted judge model keeps prompts, retrieved context, and outputs on your own hardware and makes per-output cost predictable — useful when the workflow handles internal documents or customer data. The tradeoff is judge quality: a smaller local model may agree less often with human labels than a frontier model, so the agreement check has to be re-run for each judge you consider. The pipeline shape does not change: pre-screen, sample, binary score, failing claims.
For WordPress and automation workflows — AI-assisted drafting, product copy, support replies — the same measurement applies to drafts rather than published posts. Log the brief, the source material, and the generated draft; pre-screen for numbers and links that do not appear in the source; judge a sample; record the failing claims. Keeping a human review checkpoint before publication means the rate functions as a release gate for prompt and model changes rather than a post-mortem.
A first week of measurement
- Pick one workflow and write down the unit you are scoring — one response, or one claim.
- Confirm that input, context, and output are logged on the same trace.
- Ship two or three deterministic pre-screens and log their flag rate.
- Run the judge on every flagged trace plus a random sample of about 100 unflagged ones.
- Hand-label 30 outputs and record agreement with the judge.
- Publish the rate with its sample size, the definition you used, and the date range.
If you run this on a local stack or inside a WordPress automation, share your workflow, judge model, and observed agreement in the community — the practical details of what a judge misses are exactly the kind of thing that is hard to find in vendor documentation.
FAQ
Can a hallucination rate be measured without ground truth?
Yes, for grounded workflows. Langfuse's production pattern uses a reference-free judge that checks the output against the input and context recorded on the same observation, so no known-correct answer is required. Open-domain workflows without retrieved evidence still need a reference answer, a rubric, or human review.
What is the difference between faithfulness and a hallucination rate?
Faithfulness is a per-output judgment about whether the answer is supported by its context. A hallucination rate is the aggregate share of sampled outputs flagged as unsupported. Fiddler's cookbook pairs RAGFaithfulness with AnswerRelevance and ContextRelevance so you can tell a generation failure from a retrieval failure before aggregating.
How many outputs should be sampled?
The supplied sources do not prescribe a sample size. Sampling error shrinks as the sample grows, so choose a fixed weekly sample you can afford, report the sample size next to the rate, and widen the sample as traffic volume increases.
Does a binary verdict hide how bad a hallucination is?
Yes. An invented API method and a minor unverifiable aside both score as hallucinated. Langfuse's guidance to list the failing claims in the score comment is what preserves that detail for triage.