Spend the Reranking Budget Where Candidate Quality Changes

Reference

A reranker can make retrieval better by reading the query and candidate together. It can also add latency and cost to 100 candidates when the relevant evidence was already first—or absent from the pool entirely.

I choose the reranking depth from a quality curve, not a round number.

Retrieval and reranking solve different failures

The candidate retriever aims for recall:

Find a broad set that probably contains the evidence.

The reranker aims for ordering precision:

Given this smaller set, put the most useful evidence first.

If the correct document is not in the candidate set, the reranker cannot invent it. If it is already at rank one and clearly separated, reranking 99 more items may not improve the result.

Build the candidate-depth curve first

For labelled questions, I calculate whether sufficient evidence appears by depth:

These numbers are illustrative. In this shape, reranking 200 candidates spends four times the work of 50 for almost no recall opportunity.

I also inspect what remains in the missing 3.5%. It may require better chunking, exact keyword retrieval, metadata, or corpus coverage rather than a larger reranker input.

Measure the reranker's marginal change

For each depth n, I compare:

candidate recall@n
relevant rank before reranking
relevant rank after reranking
answer evidence sufficiency
p50 and p95 reranker latency
input tokens or billed units
timeout and error rate

The useful quantity is not only final quality. It is quality gained per added candidate, millisecond, and unit of cost.

marginal value from 50 to 100
  = quality(100) - quality(50)
    versus
    latency(100) - latency(50)

The knee of this curve is a better default than “top 100.”

Query families need different budgets

An exact query such as CARGO_BUILD_WARNINGS may have one high-confidence lexical result. A broad comparative question may need several sections from different documents.

I label query slices:

exact identifier
single-fact lookup
conceptual explanation
multi-document comparison
ambiguous short query
version-sensitive query
permission-constrained query

Then I allow different candidate and context budgets when the evaluation proves a difference. Adaptive depth should use observable query features and hard maximums, not an extra model call that costs more than it saves.

Duplicates waste the budget

Fixed-overlap chunking can return five versions of the same paragraph. Reranking all five gives the appearance of depth without corpus diversity.

Before or during reranking I group:

• identical content hashes;
• overlapping ranges from one document version;
• old and new versions of the same section;
• parent and child chunks that carry the same evidence.

I preserve the best representative and keep enough diversity for alternative evidence. Deduplication must not collapse two documents that independently support an important claim.

Batch shape affects latency

A cross-encoder or model reranker may score candidates individually, in a batch, or in a listwise prompt. The maximum text per candidate and total batch size affect truncation and throughput.

I record:

candidate count
total input tokens
longest candidate
truncation rate
batch size and concurrency
provider rate limit
queue wait versus model time

Sending twice as many candidates can more than double p95 latency when it crosses a batch, token, or rate-limit boundary.

Truncation can erase the relevant part

A candidate labelled relevant may contain the answer near its end. If the reranker reads only the first 512 tokens, it is evaluating a different passage.

This returns to chunk construction: answer-sized semantic units are easier and cheaper to score. Chunk Documents by Meaning Before Tuning Chunk Size covers that boundary.

I count a candidate as visible to the reranker only when the evidence span survived its actual preprocessing.

Failure must degrade safely

The reranker can time out or reach a provider limit. I choose a fallback:

use fused retrieval order
reduce candidate depth and retry within deadline
return a bounded no-answer result
use a local cheaper reranker

The fallback never bypasses authorization filters. It also preserves source and version metadata so answer citations remain possible.

I include fallback quality in the evaluation rather than measuring only successful reranker calls.

Reranking cannot replace permissions

Only authorized candidates enter the stage. A reranker seeing forbidden text and later dropping it is already a data exposure to that model or service.

The order is:

authorize and filter
retrieve candidates
fuse and deduplicate
rerank
select evidence
generate answer

Tenant filters and document ACLs must be enforced by the retrieval data path, not expressed as natural-language instructions to the reranker.

Evaluate the final evidence too

Improving mean reciprocal rank is useful only when it improves what the answer sees. I track:

• sufficient evidence in the final context;
• missing qualifiers or warnings;
• claim-level citation correctness;
• abstention when evidence is absent;
• answer accuracy by query slice;
• latency and cost at the end-to-end deadline.

A reranker might move one relevant chunk from rank four to rank one while removing a second chunk required for a comparison. Single-relevance labels would call this an improvement; the answer becomes incomplete.

The decision record

My production choice says:

retrieve: top 60 lexical/vector candidates
deduplicate: maximum 2 overlapping chunks per document section
rerank: top 40, hard input-token cap
select: up to 8 chunks under context budget
deadline: 350 ms for reranking
fallback: fused rank order
re-evaluate: after corpus, embedding, chunker, or reranker change

The numbers come from the system's measured curve, not this example.

Reranking budget belongs where candidate quality changes. I spend enough to include and reorder useful evidence, then stop when more candidates add only latency, cost, and duplicates.