When a language model gives you a wrong answer, it rarely sounds unsure. It states the error in the same fluent, well-punctuated register it uses for facts. That mismatch — confident tone, unreliable content — is the most common reason people misjudge what an AI system can be trusted to do.
The short explanation: confidence in the output is a property of the text, not a certificate of truth. OpenAI’s September 2025 research note and the accompanying paper argue that models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty. Once you see that incentive structure, the behavior becomes predictable — and partly manageable.
What counts as a hallucination
OpenAI defines hallucinations as “plausible but false statements generated by language models.” The paper is explicit that this differs from the human perceptual experience of hallucination; in AI, it is an error mode, not a sensory event.
In practice, hallucinations look mundane. A fabricated citation with a real-looking identifier. A version number that never existed. A correct-sounding biography with the wrong birth year. An API parameter that reads like the documentation but is not in it. The defining feature is not strangeness — it is plausibility.
It is worth separating hallucination from ordinary error. A model that reports last year’s pricing because its training data is older is out of date, not fabricating. A model that invents a pricing page that never existed is hallucinating. The distinction matters because the remedies differ: outdated information is fixed by fresh retrieval, while fabrication requires verification against a primary source regardless of how current the model is.
Why training and grading push models toward guessing
The paper’s central argument is that hallucinations “originate simply as errors in binary classification.” If a model cannot distinguish a correct statement from an incorrect one, then hallucinations in a pretrained language model arise through ordinary statistical pressures rather than through any exotic failure.
Why do they persist? Because of how models are graded. The OpenAI note uses a multiple-choice analogy: if you do not know the answer but guess, you might get lucky and score; leaving it blank guarantees zero. When models are evaluated only on accuracy — the percentage of questions answered exactly right — the rational strategy is to guess rather than say “I don’t know.” The paper describes this as an “epidemic” of penalizing uncertain responses.
Two mechanisms reinforce each other. Pretraining rewards fluent continuation of text, and fluent continuation of a factual-sounding question usually looks like a factual-sounding answer. Post-training and leaderboard evaluation then reward answers over abstentions. Neither stage directly optimizes for “say nothing when unsure.”
Why the prose itself sounds certain
Confident phrasing is not a separate bug layered on top of wrong content. It is the expected output of the process:
- Fluency is the target. Hedged, uncertain text is statistically rarer in the kinds of documents models learn from than assertive declarative sentences.
- You see one sample. The model generates a single path through its probability distribution. Alternative completions — including ones that would have hedged — are not shown to you.
- Decoding settings shape risk. Higher sampling temperatures increase diversity and the chance of drifting off the most likely token, which is not necessarily the correct one. Lower temperatures make output more repetitive and predictable, which is often mistaken for more accurate.
- There is usually no uncertainty channel. Unless an API exposes token-level log probabilities or you explicitly ask the model to rate its own confidence, nothing in the interface distinguishes a well-supported answer from a guess.
None of this requires the model to have a belief state that it is misreporting. There is no internal truth meter being ignored. The model produces the continuation that its training and decoding settings make most probable, and assertive prose is what that process tends to produce.
A reported example worth studying
OpenAI’s note describes asking a widely used chatbot for the title of the PhD dissertation by Adam Tauman Kalai, one of the paper’s authors. The chatbot produced three different answers, none of them correct. Asked for his birthday, it gave three different dates, likewise all wrong.
Two details matter for practitioners. First, the failure mode was not “I don’t know” — it was confident fabrication. Second, repeating the question produced different fabrications. That instability is itself a diagnostic signal: a model that actually had the fact would tend to reproduce it.
Signals that look like confidence but are not
| What you see | What it actually indicates |
|---|---|
| Fluent grammar and clean formatting | The model learned the style of authoritative text |
| Specific numbers, dates, or citations | Plausible-looking tokens, not verified retrieval |
| “According to…” phrasing | A learned rhetorical pattern |
| Stable tone across answers | Stable style, not stable facts |
| Low temperature setting | A more likely next token, not a more likely true claim |
Practical checks you can run
- Demand the primary source, then open it. Treat every citation, URL, and quotation as unverified until you have loaded the page yourself. If the source does not exist, the claim behind it is unsupported regardless of how it was phrased.
- Re-ask in a fresh context. Put the same factual question to the model again in a new conversation or with a reworded prompt and compare. Divergence on a stable fact is a red flag, exactly as in the dissertation example above.
- Ask the model to separate knowledge from inference. Prompts such as “list what you are inferring versus what you are recalling, and name what you would need to verify this” can surface uncertainty. Treat the model’s self-reported confidence as a weak signal, not a measurement.
- Ground the answer in retrieved documents. Retrieval-augmented generation narrows the space for fabrication by putting real text in the context window. It does not eliminate the problem: the model can still misread, overstate, or blend the retrieved material with its own priors.
- Expose token-level confidence where your stack allows it. Some hosted APIs and local runtimes can return log probabilities for generated tokens. A very low-probability token inside a factual claim is a warning sign. A high-probability token is not proof of correctness.
- Put a human checkpoint before anything consequential. In a WordPress publishing workflow, that means a review step between generation and publication, plus a short record of which claims were checked and against what. Automation should prepare the draft, not ship the facts.
Where local models fit
Running a model on your own hardware does not make it more truthful. What it changes is control and observability. You choose the model and its quantization, the sampling parameters, the system prompt, and whether retrieval is in the loop. You can inspect token probabilities directly instead of relying on a provider’s summary. For teams working with sensitive material, that combination is often the real reason to run locally — not accuracy, but visibility into how an answer was produced and where the data went.
Keep the tradeoff in view: smaller and more aggressively quantized models are generally more prone to factual errors than larger ones, though the relationship is not a simple rule and depends on the task. A local model is still a model that can guess confidently.
What does not fix the problem
- “Only answer if you are sure.” This instruction helps at the margins but does not remove fabrication; the model’s sense of being sure is generated text, not a calibrated measurement.
- Adding another hallucination benchmark. The paper argues the opposite: the fix is to change how existing, dominant benchmarks are scored, because accuracy-only grading keeps rewarding guesses. Adding evaluations alongside misaligned scoring does not change the incentive.
- Assuming reasoning models are immune. OpenAI states that GPT-5 has significantly fewer hallucinations, especially when reasoning, but that they still occur. Fewer is not none.
A working stance for practitioners
Treat model output as a draft, and treat its tone as irrelevant to its truth value. The practical rule is proportional verification: the more costly a wrong claim would be, the more independently it must be checked. Grounding answers in retrieved documents and keeping a human checkpoint before publication make that verification cheap enough to actually happen.
If you are evaluating models for your own work, build a small test set from questions you already know the answers to, and score two things: whether the model gets them right, and whether it abstains when it should. That second number is the one most public leaderboards do not show you.
If you have run into a specific confident-but-wrong output, the useful detail for the community is the setup: model and version, runtime, quantization, prompt, and the exact text you got back. Sharing those makes the failure reproducible instead of anecdotal.
FAQ
Does a confident tone mean the model believes the answer?
No. There is no belief state being reported. Fluency and assertiveness are properties of the generated text, shaped by training data and decoding settings.
Will asking the model to say "I don't know" fix hallucinations?
It helps at the margins, but the paper's argument is that the incentive comes from how models are graded on accuracy. A prompt change does not alter that scoring pressure.
Does retrieval-augmented generation eliminate hallucinations?
No. It reduces them by placing real source text in the context window, but the model can still misread, overstate, or blend the retrieved material with its own assumptions.
Are reasoning models hallucination-free?
No. OpenAI reports that GPT-5 has significantly fewer hallucinations, especially when reasoning, but that they still occur.