AIForum.

How to Automate Claim Verification with RAG and Citation Retrieval

On this page

THE SHORT ANSWER

Automating claim verification means adding a third stage to a RAG pipeline: after the model answers, a separate process splits the answer into claims, matches each claim against the retrieved sources, and assigns a verdict. Documented implementations agree on the mechanics — decompose, embed, compare by similarity, then escalate ambiguous cases to a stricter entailment check.
Quick facts

Automating claim verification means adding a third stage to a retrieval-augmented generation (RAG) pipeline: after the model answers, a separate process splits the answer into individual claims, matches each claim against the retrieved sources, and assigns a verdict. The documented implementations differ in packaging but agree on the mechanics — decompose, embed, compare by similarity, then escalate the ambiguous cases to a stricter entailment check.

This guide is for practitioners who already have a RAG loop working and now need to know whether its answers are actually supported by the documents that were retrieved. It covers the pipeline stages, the verdict vocabulary used by one documented verifier, a working code shape, calibration, and where human review still has to sit.

Why retrieval alone does not verify anything

A RAG workflow consists of two components: a retrieval step that selects and filters source materials, and an LLM inference step in which those materials are packaged with the query and the model is instructed to answer from them. llmware describes this two-step process as especially effective in enterprise use cases that draw on private, contextual knowledge such as contracts, financial analyses, or customer support data.

The weakness is that nothing in the generation step proves the answer corresponds to the sources. A model can produce a fluent, well-cited paragraph that cites a document which was never retrieved, or quotes a real source for a claim that source does not make. As the AgentOS documentation puts it, LLMs hallucinate citations: they reference papers that do not exist, quote sources unrelated to the claim, or state facts the retrieval never returned — and longer answers compound the failure rate. Formatting a citation is not the same as verifying one.

The three-stage pipeline

Verification becomes a distinct stage. llmware calls it a crucial third step that is gaining increasing attention: what happens after the LLM transaction. Doing it systematically requires an integrated pipeline that captures the inputs leading up to the prompt, then integrates the inference data so you can run post-processing, source verification, ongoing review and audit, and a cycle of continuous improvement.

  1. Retrieval — select and filter source chunks.
  2. Generation — answer with the chunks in context, citing stable source identifiers.
  3. Verification — grade the answer claim by claim against those chunks.

Stage 1: make sources citable before generation

Verification compares claims to sources, so the sources must survive the round trip intact. Practical requirements:

  • Stable identifiers. Every chunk needs an ID the model can emit and your verifier can resolve back to text — a document ID plus a chunk index is usually enough.
  • Metadata alongside text. The AgentOS VerificationSource type is described as { content, title?, url? }, which is a reasonable minimum: the text being verified, plus enough provenance to show a human where it came from.
  • Log the exact chunk text that entered the prompt. If you re-chunk the corpus later, the same ID may point at different text, and old verdicts become unreproducible.
  • Keep retrieval scores. A claim supported only by a low-ranked chunk deserves more scrutiny than one supported by the top hit.

Stage 2: constrain generation, but do not trust the citations

Instruct the model to attach source IDs to specific sentences rather than to the answer as a whole. This makes the downstream split easier, but treat the model’s own citation placement as a hint, not evidence. The verifier re-derives support from the source text; it does not take the model’s word for which source backs which sentence.

Stage 3: grade claim by claim

The AgentOS CitationVerifier documents a concrete procedure. It splits the text into claims, embeds the claims and the sources in one call, scores each claim against each source by cosine similarity, and gives each claim one of four verdicts: supported, weak, unverifiable, or contradicted. The contradicted verdict comes only from an optional NLI (natural language inference) function, which is asked about the claims graded supported. The implementation lives in src/cognition/rag/citation/ and exports CitationVerifier, the VerifiedResponse and VerificationSource types, and a formatVerifiedResponse helper from @framers/agentos.

What each verdict means for automation

  • supported — the claim is close to a retrieved source. Eligible to pass, subject to your threshold calibration.
  • weak — partial or loose alignment. Route to review rather than auto-publishing.
  • unverifiable — no retrieved source covers the claim. This is not the same as false; the claim may be true but out of scope for the corpus, or it may be invented. Either way it should not be presented as sourced.
  • contradicted — a source asserts the opposite. This is the verdict that should block publication outright.

Why contradiction detection needs a separate model

Cosine similarity measures topical closeness, not entailment. “The contract renews automatically” and “the contract does not renew automatically” share almost all of their vocabulary and will embed close together. That is why the documented design asks an NLI function — not the similarity score — to produce the contradicted verdict. If you skip the NLI step, your pipeline can detect unsupported claims but not refuted ones.

A documented implementation shape

The AgentOS docs describe a one-flag path: give the agent verifyCitations, and each generate() result carries verdicts on result.grounding. The retriever returns VerificationSource objects and the embedding function is supplied by the caller.

import { agent, embedText, type VerificationSource } from '@framers/agentos';

declare function searchDocs(query: string): Promise<VerificationSource[]>;

const embedFn = async (texts: string[]) =>
  (await embedText({
    provider: 'openai',
    model: 'text-embedding-3-small',
    input: texts,
  })).embeddings;

const docsAgent = agent({
  provider: 'openai',
  model: 'gpt-4o',
  verifyCitations: {
    embedFn,
    retrieve: (query) => searchDocs(query),
  },
});

const result = await docsAgent.generate('How do I configure a guardrail?');
console.log(result.text);
console.log(result.grounding?.overallGrounded);

The observable check after wiring this up is result.grounding?.overallGrounded plus the per-claim verdicts. If grounding is undefined, the verifier did not run — check that verifyCitations was actually passed to the agent and that the retriever returned at least one source.

llmware’s framing: verification as an audit loop

llmware approaches the same problem from the operations side. Its demonstration builds a RAG workflow that performs basic contract analysis and then verifies the accuracy of the LLM response against the sources. The emphasis is on scale: with potentially a large volume of LLM transactions, the question is how to systematically verify that responses correspond to specific sources with citations. Its answer is to capture the inputs leading up to the prompt and integrate the inference data afterwards, which is what makes ongoing review, audit, and continuous improvement possible at all.

The practical takeaway from both sources is the same: verification is only as good as your record of what was retrieved. If you cannot reconstruct the exact prompt-time context, you cannot re-run a verdict or defend it to an auditor.

Calibration is the part you have to do yourself

Neither source publishes universal similarity thresholds, and none should — the right cutoff depends on your embedding model, your chunk sizes, and how much risk you tolerate. The workable approach is a small labeled set:

  1. Collect 30–50 real answers from your pipeline.
  2. Split each into claims and label every claim-source pair as supported, weak, unverifiable, or contradicted by hand.
  3. Sweep the similarity threshold and record how many pairs move between supported and weak.
  4. Pick the threshold that keeps false supported verdicts acceptably low, then accept the review queue size that results.

Tighten the threshold and the review queue grows; loosen it and unsupported claims slip through as supported. There is no setting that removes the tradeoff, only one that matches your tolerance.

Cost, latency, and local options

Verification adds work per answer: one batched embedding call covering the claims and the sources, plus NLI calls for the claims graded supported. That is cheaper than a second full generation pass, but it is not free, and it scales with answer length because longer answers produce more claims.

For local deployments, the documented interface is a reason for optimism: embedFn is a function the caller supplies, and the NLI step is likewise an optional function rather than a hard-coded service. That means the embedding and entailment stages can be pointed at locally hosted models when document text must not leave the machine. This is a property of the documented API shape, not a tested local configuration — verify it against your own runtime before relying on it.

Where human review still belongs

Fully automatic verification is a routing decision, not a replacement for review. A defensible default:

  • contradicted → block, and surface the conflicting source pair to a human.
  • weak or unverifiable → hold as draft, queue for review.
  • supported → eligible for the next automated step, with verdicts stored alongside the output.

For WordPress and other publishing workflows, storing verdicts as post metadata is what makes the gate useful later: if the post body is edited after verification, the stored verdicts no longer describe the current text, and the check should be re-run before publication rather than trusted.

Known limitations

  • Scope is the retrieved set. A true claim whose supporting document was never retrieved is graded unverifiable. Verification detects unsupported answers; it does not fix retrieval.
  • The NLI model has its own error modes. Contradiction detection inherits the false positives and false negatives of whatever entailment model you supply.
  • Source authority is out of scope. A claim can be faithfully supported by a retrieved document that is itself outdated or wrong.
  • Long answers degrade. The AgentOS docs note that longer answers compound the citation failure rate, so verification load and error surface both grow with output length.

A reasonable first step

Pick one existing RAG endpoint, log the retrieved chunks and their IDs alongside every answer for a week, then hand-label claims on 30–50 of those answers. That labeled set tells you whether your current retrieval is even capable of supporting the claims your model makes, and it gives you the calibration data you need before you automate any routing decision.

If you try this on your own stack, share your environment, embedding model, threshold, and the verdict distribution you observed in the community — concrete numbers from different corpora are more useful than another architecture diagram.

FAQ

Does cosine similarity alone detect contradictions?

No. In the documented AgentOS design, the contradicted verdict comes only from an optional NLI function, which is asked about the claims graded supported. Similarity scoring alone can flag weak or unsupported claims, but not refuted ones.

What is the difference between citation retrieval and citation verification?

Citation retrieval supplies sources to the model and asks it to cite them. Citation verification runs afterwards, splitting the answer into claims and grading each claim against the retrieved source text to check that the citation is actually warranted.

Can claim verification run with local models?

The documented AgentOS interface takes a caller-supplied embedFn and an optional NLI function rather than hard-coding a provider, so the embedding and entailment stages can in principle point at locally hosted models. Treat this as a property of the API shape and verify it against your own runtime before depending on it.

What should happen to a claim graded unverifiable?

Unverifiable means no retrieved source covers the claim; it does not mean the claim is false. In an automated flow it should be held for review rather than published as sourced, because the pipeline cannot distinguish a true out-of-corpus fact from an invented one.

References

  1. https://llmware.ai/resources/techniques-for-automated-source-citation-verification-for-rag
  2. https://docs.agentos.sh/features/citation-verification

Related terms

RAG (retrieval-augmented generation)
A two-step architecture in which source materials are retrieved, then packaged with the query so the LLM answers from them.
Claim decomposition
Splitting a generated answer into individual assertions so each can be graded separately against sources.
Cosine similarity
A distance measure between embedding vectors, used to score how close a claim is to a source passage.
NLI (natural language inference)
A model task that judges whether one statement entails, contradicts, or is neutral toward another; used here to produce the contradicted verdict.
Grounding
The property of an answer being supported by the retrieved context; AgentOS exposes it as result.grounding.
Hallucinated citation
A citation that points at a nonexistent source, or at a real source that does not support the attached claim.

Leave a Reply

Your email address will not be published. Required fields are marked *