Integration · Lesson 5 of 8

1.5 - Designing a RAG Pipeline: Chunking and Indexing

Design a RAG ingestion, chunking and indexing pipeline, use contextual enrichment and Claude's citations feature appropriately, and diagnose chunking failures from their symptoms.

A retrieval-augmented generation (RAG) pipeline can only answer from what it retrieves, and what it can retrieve was decided at ingestion time, when nobody was looking at a user's question. Chunking and indexing choices therefore set a ceiling on answer quality that no prompt or model change can lift. First decide whether you need RAG at all: Anthropic's Contextual Retrieval post suggests that a knowledge base under about 200,000 tokens (roughly 500 pages) can simply be placed in the prompt with prompt caching. That figure is from the post, so re-check it against current context limits and caching behaviour. Retrieval strategy is Lesson 1.6; this lesson covers ingestion, chunking and indexing.

From raw documents to an index

Parse and clean. Extract text with its structure (headings, tables, code), strip boilerplate, and dedupe near-duplicate documents. Scanned PDFs need OCR first; note that Claude's citations feature cannot cite a PDF that has no extractable text.
Chunk along meaningful boundaries (next section).
Enrich. Attach metadata: source ID, title, section path, version or date, language, and the access-control or tenant attributes that retrieval will filter on. Optionally add generated context (below).
Embed. Anthropic does not offer its own embedding model; its docs point to Voyage AI and recommend setting input_type to distinguish documents from queries. Vectors from different models or dimensions are not comparable, so changing embedding model means re-embedding the whole corpus.
Index. Store chunk text and metadata with the vectors, and add a keyword (BM25) index if you plan hybrid retrieval.
Keep it fresh. Update incrementally by document ID and content hash, delete or tombstone removed documents, version them, and set a staleness target you can monitor.

Chunking decisions and their trade-offs

Decision	Risk if too small or absent	Risk if too large or heavy
Chunk size	Chunks lose the context to be interpretable, and answers span several chunks	Embeddings blur several topics, tokens are wasted, and the answer is buried
Boundaries	Fixed-size cuts split sentences, tables and functions	Whole documents as chunks defeat precision
Overlap	Facts at a boundary are lost	Duplicated content inflates the index and fills top-k with near-copies
Metadata	No filtering, no attribution, no freshness signals	Inconsistent fields that nothing uses

Prefer structure-aware boundaries (headings, paragraphs, table rows with their headers, code functions) over fixed token counts, and treat overlap as a patch rather than a design. A common pattern is parent-child chunking: match on small chunks for precision, then pass the enclosing section to the model for context.

The context problem has a documented fix. Anthropic's Contextual Retrieval prepends a short, chunk-specific explanation (typically 50–100 tokens) generated by Claude to each chunk before embedding and BM25 indexing, so "revenue grew by 3% over the previous quarter" becomes attributable to a company and period. In Anthropic's experiments, top-20 retrieval failure fell from 5.7% to 3.7% with contextual embeddings, 2.9% when combined with contextual BM25, and 1.9% with reranking. Those are results on their datasets; measure yours. Prompt caching keeps per-chunk generation affordable because the document is reused, and the embeddings docs also list Voyage models that produce contextualised chunk embeddings without manual augmentation.

Key concept: retrieval is the ceiling

If the evidence is not in the retrieved set, generation cannot recover it. Evaluate retrieval separately: for a labelled set of questions, is the gold chunk in the top-k? When it is not, the problem is chunking, indexing or retrieval, and swapping the generation model will not help.

Citations: making chunks attributable

Claude's citations feature grounds answers in your documents. You set citations: {enabled: true} on document blocks; plain text and PDFs are automatically chunked into sentences, while custom content documents use your content blocks as-is with no further chunking. Anthropic's guidance for RAG is to put each chunk in a plain-text document if you want sentence-level citations, or use custom content if you want to control granularity. Citations come back as char_location, page_location or content_block_location objects with a cited_text field that does not count toward output tokens. Constraints: citations must be enabled on all or none of the documents in a request, and combining them with structured outputs returns a 400 error.

For tool-based retrieval, search_result blocks (with source, title, content and citations) can be returned from a tool or supplied as top-level content, and Claude cites them automatically. In a tool result, if any block is a search_result then all must be. The design consequence: your chunk boundaries and stable source identifiers decide citation granularity and what a user can verify.

How chunking failures show up

Symptom	Likely cause	Direction of fix
Right topic, wrong document or entity	Chunk lacks its context ("the company", "Section 4.2")	Contextual enrichment; prepend title and section path
Partial or wrong answers where facts sit in adjacent chunks	Boundary splits, chunks too small	Structure-aware boundaries; parent-child
The right chunk exists but is never retrieved	Diluted large chunks, low top-k, or an exact-term miss	Smaller chunks; rerank; hybrid retrieval (Lesson 1.6)
Top-k full of near-duplicates	Heavy overlap, duplicated sources	Dedupe; reduce overlap
Wrong numbers from tables	Table split, headers lost	Keep tables whole or repeat headers
Outdated answers	No update or delete path	Incremental re-index, versions, freshness metadata
Users see content they should not	Access metadata missing or applied after retrieval	Filter at retrieval by the user's entitlements (Lesson 1.2)

Common exam distractor

"Larger context windows make chunking irrelevant" ignores cost, latency and dilution. "Increase chunk size to restore missing context" blurs embeddings without adding document-level context; enrichment does that. "Generate embeddings with Claude" is wrong because Anthropic offers no embedding model. And "switch to a bigger generation model" is the classic wrong-layer fix when the gold chunk was never retrieved.

Exam traps

Assuming very large context windows make chunking and indexing strategy unimportant

Retrieved text still costs tokens and latency, and irrelevant or diluted context can hurt answers. Corpora also exceed any window. Chunking determines what enters the window and whether the right evidence is retrievable at all.
Fixing missing context by simply making chunks larger

Larger chunks blur the embedding and waste tokens, and they still may not name the document or section. Prepending chunk-specific context (contextual retrieval) or metadata addresses the actual loss.
Using a Claude model to produce the embeddings for the vector index

Anthropic's docs state it does not offer its own embedding model and point to third-party providers such as Voyage AI. Claude can generate chunk context and answer from retrieved text, but embeddings come from an embedding model.
Diagnosing wrong answers by tuning the prompt or upgrading the generation model before checking retrieval

If the gold chunk never reaches the context, no generation change can recover it. Measure recall of the gold chunk in the top-k first to decide whether the fault lies in chunking, indexing, retrieval or generation.

Practice question

A legal-operations team built a policy assistant over 400 HR and finance policy documents, split into fixed 200-token chunks. Users report answers that cite the wrong policy: a query about contractor expense limits returns a chunk beginning 'Section 4.2: Exceptions apply within 30 days', which belongs to a different policy. Recall of the correct chunk in the top 10 is low on the team's labelled questions. What is the best first change?

A Move to a larger generation model so it can work out which policy each retrieved chunk belongs to, instruct it to name the policy in every answer, and re-run the labelled questions to compare answer accuracy
The model only sees what is retrieved, and a chunk without its policy name is ambiguous to it too. The measured problem is low recall, so this changes the wrong layer.
B Re-chunk on heading boundaries, prepend each chunk's policy title and section path (or generated context), store policy ID and version as metadata, and re-measure recall in the top-k Correct
It addresses the diagnosed cause, chunks that lack the context needed to be retrieved and interpreted, and adds metadata for filtering and attribution. It also verifies the change against the labelled retrieval metric.
C Raise chunk overlap to 50% so every chunk repeats half of its neighbour, and re-index the whole corpus so boundary sentences are always preserved
Overlap does not add policy-level context to a chunk, and heavy overlap fills the top-k with near-duplicates and inflates the index.
D Retrieve the top 50 chunks instead of the top 10 and let the model choose, adding a prompt line to ignore chunks from unrelated policies
It may raise recall somewhat, but it costs tokens and latency and leaves the ambiguous chunks ambiguous. It does not fix the root cause and can add distractors.

Build exercise: Chunk, enrich, evaluate and cite a small knowledge base

Intermediate · 120 minutes

You'll practice:

Design an ingestion, chunking, metadata and freshness plan for a document set
Compare chunking variants using recall of gold chunks rather than intuition
Add generated context to chunks and measure whether it helps
Return retrieved chunks to Claude in a form that supports verifiable citations
Map observed retrieval symptoms to chunking or indexing causes

Pick a set of 10 to 20 documents you can share (for example a policy handbook or product docs). Write the ingestion plan: parsing approach, cleaning and dedupe rules, metadata fields, and the update and delete strategy.

Ingestion decisions are invisible in production until they fail. Writing the plan first exposes missing metadata such as access attributes and versions.

You should see: A one-page plan listing each stage, the metadata schema, and how a changed or deleted document propagates to the index.
Hints
1. ```
Which metadata will retrieval need to filter on, and which will the citation need to show a user?
```
2. ```
Include source_id, title, section_path, version or date, and an access attribute. Key updates on source_id plus a content hash so unchanged documents are skipped.
```
3. ```
Schema example: {chunk_id, source_id, title, section_path, version, acl, text}. Update rule: on document change recompute its hash; if different, delete all chunks with that source_id and re-insert.
```

Implement two chunkers over the same documents: fixed 200-token windows with 20% overlap, and structure-aware chunks split on headings with the section path stored as metadata.

You cannot judge chunking without a comparison on identical data, and the structure-aware variant is the baseline the docs and post imply.

You should see: Two chunk lists with counts, average size and a sample of ten chunks from each.

Hints

What happens to a table or a numbered procedure in each variant?

Split markdown on lines starting with # and record the heading stack as section_path. Cap very long sections by splitting on paragraphs, never mid-sentence.

def chunk_by_heading(md, source_id):
    chunks, path, buf = [], [], []
    for line in md.splitlines():
        if line.startswith('#'):
            if buf: chunks.append({'source_id': source_id, 'section_path': ' > '.join(path), 'text': '\n'.join(buf)}); buf = []
            level = len(line) - len(line.lstrip('#')); path = path[:level-1] + [line.lstrip('# ').strip()]
        buf.append(line)
    if buf: chunks.append({'source_id': source_id, 'section_path': ' > '.join(path), 'text': '\n'.join(buf)})
    return chunks

Write 15 or more questions and mark the gold chunk (or chunks) for each. Build a retriever, using a vector index with an embedding model of your choice or BM25, and compute recall of the gold chunk in the top 5 and top 10 for each chunking variant.

This separates retrieval quality from generation quality and turns the chunking choice into evidence.

You should see: A small table: variant, recall@5, recall@10, plus the list of questions that failed in each.

Hints

For the failed questions, does the retrieved chunk name the document or entity the question asked about?

Store gold chunk IDs from a stable source_id plus section_path so they survive re-chunking. Compute recall as the fraction of questions where any gold chunk appears in the top-k.

recall_at_k = sum(1 for q in qs if set(q['gold']) & set(top_k(q['question'], k))) / len(qs). Because chunk boundaries change between variants, match gold by section_path or source_id rather than chunk_id.

Add generated context: for each chunk, ask Claude for a 50 to 100 token description that situates the chunk within its document, prepend it before indexing, and re-run the recall measurement.

This tests Anthropic's contextual retrieval idea on your data instead of taking published numbers on trust.

You should see: Recall for a third variant, and a few side-by-side examples of chunks before and after enrichment.

Hints

How would you avoid paying to send the full document with every chunk?

Put the document in a cached prefix and vary only the chunk in the final message. Instruct the model to answer only with the context and nothing else.

prompt = f'<document>\n{doc_text}\n</document>\nHere is a chunk from that document:\n<chunk>\n{chunk_text}\n</chunk>\nWrite 1-2 sentences situating this chunk within the document (which document, which section, which entity) to help search retrieval. Output only that context.'
Add cache_control to the document block so repeated calls for the same document reuse it.

Send the top retrieved chunks to Claude as a custom content document with citations enabled, ask a question, and verify that each citation maps back to a chunk ID and that cited_text is present.

Citations are only useful if you can trace them back to a source unit and show it to a user. The mapping is your chunk design's payoff.

You should see: A response whose text blocks carry content_block_location citations with start_block_index and end_block_index that you can convert to chunk IDs.

Hints

What does the start_block_index refer to in your list of chunks?

Build one document whose content array holds one text block per retrieved chunk, in order, and keep a parallel list of chunk IDs. Remember that structured outputs cannot be combined with citations.

doc = {'type': 'document', 'source': {'type': 'content', 'content': [{'type': 'text', 'text': c['text']} for c in top]}, 'title': 'Retrieved context', 'citations': {'enabled': True}}
After the call, for each citation of type content_block_location use top[cit.start_block_index]['chunk_id'] to show the source.

Sources

Introducing Contextual Retrieval (Anthropic) - Anthropic
Citations (Claude API docs) - Anthropic
Search results content blocks (Claude API docs) - Anthropic
Embeddings (Claude API docs) - Anthropic
Effective context engineering for AI agents (Anthropic Engineering) - Anthropic