- Published on
Citations Need Claim-Level Provenance, Not Decorative Links
- Authors

- Name
- Mehdi Akiki
Reference
Adding three links below an AI answer does not make the answer grounded.
A reader needs to know which source supports which claim. The system needs to know the exact source version and passage used. The evaluator needs to distinguish “retrieved nearby” from “actually supports this statement.”
I model citations as claim-to-evidence edges, not as decorations attached to a whole response.
One answer can contain several evidence states
Consider:
The service retains audit events for 90 days.
Exports are available to administrators.
The policy changed because storage costs increased.
One document may support the first sentence, another the second, and nothing may support the explanation in the third.
A list of both documents at the bottom makes the unsupported explanation look sourced.
I split the answer into material claims:
C1 retention duration -> source A, passage 3
C2 export permission -> source B, passage 8
C3 reason for change -> unsupported
The third claim must be removed, qualified as an inference, or sent for another retrieval step.
Provenance needs stable identities
My smallest useful representation is:
type Evidence = {
sourceId: string;
sourceVersion: string;
canonicalUri: string;
contentHash: string;
selector: {
start: number;
end: number;
exactQuote: string;
prefix?: string;
suffix?: string;
};
retrievedAt: string;
};
type ClaimSupport = {
claimId: string;
claimText: string;
evidenceIds: string[];
relation: "supports" | "contradicts" | "context_only";
verdict: "verified" | "partial" | "unsupported" | "conflicting";
};
The W3C PROV ontology provides a general model for entities, activities, agents, and derivation. I do not need to expose PROV-O directly in every API, but its distinction between a result, the activity producing it, and the source entities is useful.
A URL does not identify the evidence used
Web pages change. Documents are replaced at the same path. A database row has no public URL. One 80-page PDF can contain both supporting and contradicting passages.
I store:
logical source identity
version, revision, or retrieval-time content hash
exact passage selector
document title and canonical URI for display
access-control context
retrieval and transformation lineage
For text, I combine position with a quote and nearby context. The W3C Web Annotation model defines both text-position and text-quote selectors. Position is efficient; quote plus prefix/suffix can help re-anchor after small edits.
If the version has materially changed, I mark the citation stale instead of pointing confidently to new text at the old URL.
Chunk IDs are not enough
An internal chunk ID locates one retrieval artifact. It may depend on parser and chunker versions and disappear after reindexing.
I keep both:
retrieval identity: chunk and index generation used for this answer
source identity: document version and source range that can be verified again
A Retrieval Index Needs an Invalidation Strategy explains why chunk identities change when content, parsing, permissions, or embedding configuration changes.
The answer provenance should survive an index rebuild even if its evidence later becomes stale.
Retrieval does not prove support
A retriever returns passages related to a query. It does not prove that a passage entails a generated claim.
These failures are common:
source mentions 90 days for logs; answer applies it to all records
source says administrators may export; answer says only administrators can export
source describes an old product version; answer uses present tense
source reports a proposal; answer states it as implemented
source says not supported; lexical match drops the negation
After generation, I run a claim-support step. It can use rules, a model, or human review depending on risk, but it works from the exact claim and exact passage.
The verdict is retained. Hiding an unsupported sentence from the UI without recording the failure loses useful evaluation data.
Separate quotation, paraphrase, and inference
A short direct quotation needs exact text and appropriate quotation handling. A paraphrase needs semantic support. An inference needs its premises and an explicit label.
quoted: source contains these exact words
paraphrased: source expresses the same material fact
inferred: conclusion follows from named evidence and system reasoning
I avoid making an inference look like a quotation-backed fact. For high-stakes output, unsupported inference may be disallowed entirely.
Conflicting sources need representation
Grounding is not always one source and one truth. Two current policies, provider responses, or database records may disagree.
type ClaimEvidenceSet = {
supporting: string[];
contradicting: string[];
superseded: string[];
unresolved: string[];
};
The answer can state the conflict, prefer an authoritative source by documented policy, or decline to resolve it. Silently citing only the convenient passage hides a data-quality problem.
Source authority is domain-specific. Freshness alone does not make a blog post override a signed policy.
Permissions travel with provenance
A citation can leak a restricted title, URI, quote, or even the existence of a document. I authorize both retrieval and citation rendering for the current caller.
Filter Before Retrieval: Permissions Are Part of Relevance develops this boundary.
When an answer is shared, exported, or cached, its evidence may need re-authorization for the new reader. Permission at generation time is not a permanent public licence.
Render citations where the claim appears
I prefer a citation marker immediately after the supported sentence or paragraph. The details view shows:
source title
exact supporting passage with limited context
version or last-updated time
link when available and authorized
support status
If several consecutive claims use different evidence, one marker at the end of a long paragraph is ambiguous. If a single citation supports the whole compact paragraph, repeating it after every sentence adds noise. The data model stays claim-level even when presentation groups markers.
A provenance verification test
My fixture contains:
one passage that directly supports a fact
one topically related passage that does not support it
one passage with negation
two source versions with changed numbers
one conflicting source
one restricted source
one answer sentence joining two claims
I assert:
- every material factual claim has an ID;
- the selector resolves against the recorded source version;
- the quote matches the selected text;
- each support relation is verified separately;
- negation and scope are preserved;
- conflicting evidence remains visible to policy;
- stale versions are detected;
- unauthorized citations and metadata do not render;
- unsupported claims are removed, qualified, or reported.
For evaluation, I measure citation precision, claim coverage, selector resolution, stale-source rate, and unsupported material claims. “Percentage of answers containing a link” is not a grounding metric.
My production checklist
Before I call an answer cited, I ask:
Can I identify each material claim?
Which exact passage supports it?
Which source version was used?
Does the passage entail the full claim, including scope and time?
Are contradictions represented?
Can the current reader access the citation metadata and text?
Will the selector survive or detect source change?
Can I reproduce how retrieval, reranking, and generation used it?
A citation is useful when it lets a reader inspect evidence and lets the system recheck its own claim. A decorative URL list can create confidence. Claim-level provenance creates accountability.