Introduction
Welcome back!!! If you have been following my RAG series, you already know that chunking is where most retrieval pipelines quietly break. You split a document into neat little pieces, embed each one, store it in a vector database, and assume the system now understands your content. It does not. Each chunk sits alone, stripped of the paragraph before it and the paragraph after it, and that missing context is exactly where your RAG system starts failing.
In September 2024, Anthropic published a technique called Contextual Retrieval that tackles this problem directly. It is simple, it is cheap to run, and it cut retrieval failures by 49%, and by 67% when combined with reranking. That is not a small improvement. That is the difference between a RAG system users trust and one they quietly stop using.
Related article: Chunking Strategies
The Real Problem With Chunking
Say you are building a RAG system over a company’s financial filings. One chunk says:
“The company’s revenue grew by 3% over the previous quarter.”
Sounds fine until someone asks “What was ACME Corp’s Q2 2024 revenue growth?” Your retriever has no idea this chunk is even about ACME Corp, let alone Q2 2024. The company name and the quarter were mentioned two paragraphs earlier, in a chunk that got split away during preprocessing. The embedding for this chunk carries none of that information, so it barely resembles the query, and it gets buried under chunks that are only superficially related.
This is not a rare edge case. It is the default behavior of naive chunking, and it is the single biggest reason RAG systems return “I could not find relevant information” or, worse, confidently hallucinate an answer.
What Contextual Retrieval Actually Does
The fix Anthropic proposed is almost embarrassingly simple: before you embed or index a chunk, prepend a short piece of context that situates it within the full document. Instead of embedding
“The company’s revenue grew by 3% over the previous quarter.”
you embed
“This chunk is from an SEC filing on ACME Corp’s Q2 2024 performance. The previous quarter’s revenue was $314 million. The company’s revenue grew by 3% over the previous quarter.”
Now the chunk carries the company name, the time period, and the surrounding numbers, all inside the text that gets embedded and indexed. Both your vector search and your keyword search suddenly have something real to match against.
The two components that make this work are:
Contextual Embeddings Every chunk gets a short, chunk specific context written by an LLM before embedding. This context is generated using the whole document as reference, so it captures things like the document’s subject, the section it belongs to, and any entities or dates that got separated from the chunk during splitting.
Contextual BM25 The same contextualized chunk is also indexed for keyword search using BM25. This matters because embeddings alone still miss exact matches like order numbers, product codes, or specific dates, the same limitation we covered in the keyword search article in this series. Contextual BM25 gives you that exact match capability on top of contextualized text.
Run both together in a hybrid retriever, and you get chunks that are both semantically rich and precisely searchable.
Why This Works Better Than Alternatives
You might be thinking, why not just make chunks bigger so they carry more context naturally(That was my first thought too 🤔).A few reasons this does not hold up:
Bigger chunks dilute relevance. A 2000 token chunk covering five different topics gets a muddy embedding that does not represent any single topic well. Precision drops even as recall might improve slightly.
Bigger chunks blow up your context window. If you retrieve the top 10 chunks and each one is huge, you are burning tokens on irrelevant surrounding text instead of giving the LLM more distinct, relevant pieces of information.
Contextual retrieval keeps chunks small and precise, and adds context as metadata baked into the text itself, rather than solving the problem by brute force. You get the recall benefits of bigger chunks without the noise.
Implementation, Step by Step
Here is a working pipeline you can adapt to your own documents.
Step 1: Generate Context for Each Chunk
from anthropic import Anthropic
client = Anthropic()
CONTEXT_PROMPT = “””<document>
{full_document}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk_content}
</chunk>
Please give a short, succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.”””
def generate_chunk_context(full_document, chunk_content):
response = client.messages.create(
model=”claude-opus-4-8″,
max_tokens=150,
messages=[{
“role”: “user”,
“content”: CONTEXT_PROMPT.format(
full_document=full_document,
chunk_content=chunk_content
)
}]
)
return response.content[0].text
If you are worried about cost, Claude’s prompt caching handles this well. You cache the full document once and reuse that cached version across every chunk in that document, so you are only paying full price for the document tokens a single time.
Step 2: Build the Contextualized Chunks
def build_contextualized_chunks(document_text, chunks):
contextualized_chunks = []
for chunk in chunks:
context = generate_chunk_context(document_text, chunk)
contextualized_chunk = f”{context}nn{chunk}”
contextualized_chunks.append(contextualized_chunk)
return contextualized_chunks
Step 3: Index for Both Embedding Search and BM25
from rank_bm25 import BM25Okapi
import numpy as np
def build_hybrid_index(contextualized_chunks, embedding_model):
# Embedding index
embeddings = embedding_model.encode(contextualized_chunks)
# BM25 index
tokenized = [chunk.lower().split() for chunk in contextualized_chunks]
bm25 = BM25Okapi(tokenized)
return embeddings, bm25
Step 4: Retrieve With Both, Then Rerank
def hybrid_retrieve(query, chunks, embeddings, bm25, embedding_model, top_k=20):
# Semantic search
query_embedding = embedding_model.encode([query])[0]
semantic_scores = np.dot(embeddings, query_embedding)
# Keyword search
tokenized_query = query.lower().split()
bm25_scores = bm25.get_scores(tokenized_query)
# Combine, normalize both to 0-1 range first
semantic_norm = semantic_scores / (semantic_scores.max() + 1e-8)
bm25_norm = bm25_scores / (bm25_scores.max() + 1e-8)
combined = 0.5 * semantic_norm + 0.5 * bm25_norm
top_indices = combined.argsort()[-top_k:][::-1]
return [chunks[i] for i in top_indices]
For the final accuracy boost, add a reranker (Cohere’s rerank model or a cross encoder) on top of these top 20 results before passing the final 3 to 5 chunks to your LLM. This is the step that took Anthropic’s numbers from 49% failure reduction to 67%.
Best Practices for Contextual Retrieval
Use a cheap, fast model for context generation. You do not need your most expensive model to write a two sentence summary of where a chunk sits in a document.Cache the full document when generating context for multiple chunks from the same source. This is where prompt caching saves real money at scale.Keep the generated context short, 50 to 100 tokens is usually enough. Long context defeats the purpose of keeping chunks small.Always pair contextual embeddings with contextual BM25. Using only one leaves accuracy on the table.Add a reranking step if your use case can tolerate the extra latency. The accuracy gain is significant.Test on your own domain before trusting published numbers. Anthropic’s benchmarks used codebases, fiction, ArXiv papers, and science papers, your documents may behave differently.Monitor your actual retrieval failure rate before and after, do not assume the improvement transfers without measuring it.
Conclusion
Contextual retrieval is one of those techniques that feels obvious once you see it, and that is usually a sign it is worth adopting. It does not require a new vector database, a new architecture, or months of engineering work. It requires rethinking one step in your pipeline: what actually gets embedded and indexed.
If your RAG system has been giving vague or wrong answers and you have already tuned your chunk size, tried different embedding models, and added reranking, this is very likely the missing piece.
Thank you for following this RAG tutorial series! If you found this helpful, please give it a clap 👏 and share with others building RAG systems.
Contextual Retrieval: Anthropic’s Method for Cutting RAG Failures was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.