Chunk Documents by Meaning Before Tuning Chunk Size

Reference

Many retrieval pipelines begin with one number: 500 tokens per chunk with 50 tokens of overlap.

The number is easy to configure. It does not tell whether the chunk contains one complete idea, half of a warning, a table without headers, or code separated from its explanation.

I choose semantic boundaries first and use token size as a constraint second.

The retrieval unit should be able to answer something

Suppose a runbook contains:

## Rotating the signing key

### Preconditions
Confirm the secondary key is active.

### Procedure
1. Switch verification to accept both keys.
2. Deploy the new primary key.
3. Wait for the maximum token lifetime.
4. Remove the old key.

> Warning: removing the old key before step 3 invalidates active sessions.

A fixed window may return steps 2–4 without the precondition or warning. Every sentence is topically related, but the answer is unsafe.

The natural answer unit is the procedure with its heading, precondition, and warning.

Parse structure before counting tokens

For structured documents I extract a tree:

document
  -> section path
     -> paragraphs
     -> list or procedure
     -> code block
     -> table with header
     -> callout or warning

Then I group adjacent nodes that belong together. Only when a group exceeds the model or retrieval budget do I split it further.

I avoid splitting:

• a code fence from the paragraph that explains it;
• a table body from its header;
• a numbered procedure in the middle of one step;
• a warning from the operation it qualifies;
• a definition from its constraints;
• a question from its answer.

For plain text without reliable structure, paragraph and sentence boundaries are still better initial candidates than arbitrary token positions.

Inherit enough context into every chunk

A subsection titled “Timeout” is meaningless without its ancestors. I store a contextual header:

Document: Payments API Operations
Section: Webhooks > Delivery retries > Timeout

The chunk record can be:

type Chunk = {
  chunkId: string;
  documentId: string;
  documentVersion: string;
  sectionPath: string[];
  text: string;
  sourceStart: number;
  sourceEnd: number;
  contentType: "prose" | "procedure" | "code" | "table";
  accessPolicyId: string;
};

The inherited path improves retrieval for words absent from the local paragraph and gives the answer a useful citation label.

I keep exact source offsets separately so added context does not pretend to be verbatim source text.

Overlap cannot repair a bad boundary

Overlap helps when an idea crosses a chosen split. It also duplicates index content, increases storage, and can return several near-identical passages.

If a warning is 300 tokens away from the step it controls, a 50-token overlap does nothing. If every chunk overlaps by half, retrieval may spend its candidate budget on copies of one paragraph.

I add overlap only at the lower-level split of an oversized semantic unit. The parent section context remains explicit.

Large and small chunks fail differently

Small chunks can:

• lose definitions and qualifiers;
• match an ambiguous word without its topic;
• separate causes from consequences;
• produce many near-duplicate candidates.

Large chunks can:

• contain several unrelated answers;
• dilute embedding similarity;
• waste the model context window;
• make precise citation difficult;
• hide one relevant sentence inside much noise.

There is no universal token count because document genres differ. API reference, legal policy, incident timeline, source code, and tutorial prose have different natural units.

Use a hierarchy for long units

A long chapter may be valuable as context but too large as one retrieval result. I create levels:

document summary
  -> section summary and metadata
     -> answer-sized child chunks

Retrieval can find a section from a broad query, then search or rerank its children. The final evidence still points to the precise child source range.

I do not let generated summaries replace source evidence. They are routing aids and need invalidation when the document changes.

Tables need row context

A table such as:

cannot be safely split into a row containing only 429 | yes. Each row inherits the column names, table caption, section path, and relevant notes.

For a large table I may index row groups as text:

Status 429. Retry: yes. Meaning: caller exceeded a rate limit.
Context: Payments API > Errors > Retry policy.

The renderer can cite the original table range.

Code needs language structure too

For source code I prefer symbol-aware units: function, type, implementation block, or configuration object. A method may need its enclosing type name and imports as metadata.

Very large functions require internal splits, but each child retains:

repository and revision
file path
symbol path
language
line range
enclosing signature

Copying the entire repository into equal token windows makes definitions and usages difficult to distinguish.

Evaluate retrieval before answer generation

I create questions with known evidence locations and measure whether retrieval returns a sufficient chunk.

The labels include failure reason:

missed document
right document, wrong section
right section, missing qualifier
table header absent
code explanation separated
permission-filtered result absent
near-duplicate candidates crowded out evidence

Metrics such as recall at k are useful, but a hit is counted only when the returned unit contains enough evidence to answer correctly.

Evaluate Retrieval Separately From Answer Generation gives the wider evaluation split.

Tune size after the failure categories are visible

Once semantic parsing works, I test token caps and candidate counts by document type. For each configuration I record:

evidence recall@k
complete-answer-unit rate
duplicate-candidate rate
retrieved tokens per query
reranker latency
answer citation correctness

A smaller average chunk may improve exact lookup while hurting procedures. I keep type-specific policies when the evidence supports them.

Chunking is an information architecture decision. The best chunk is not the one that fits a fashionable number. It is the smallest self-contained unit that preserves the meaning needed to answer, with enough metadata to retrieve, authorize, update, and cite it correctly.