Welcome back!!! If you have been following my RAG series, you already know that chunking is where most retrieval pipelines quietly break. You split a document into neat little pieces, embed each one, store it in a vector database, and assume the system now understands your content. It does not. Each chunk sits alone, stripped of the paragraph before it and the paragraph after it, and that missing context is exactly where your RAG system starts\u00a0failing.<\/p>\n
In September 2024, Anthropic published a technique called Contextual Retrieval that tackles this problem directly. It is simple, it is cheap to run, and it cut retrieval failures by 49%, and by 67% when combined with reranking. That is not a small improvement. That is the difference between a RAG system users trust and one they quietly stop\u00a0using.<\/p>\n
Related article: Chunking Strategies<\/a><\/p>\n Say you are building a RAG system over a company\u2019s financial filings. One chunk\u00a0says:<\/p>\n \u201cThe company\u2019s revenue grew by 3% over the previous quarter.\u201d<\/p>\n Sounds fine until someone asks \u201cWhat was ACME Corp\u2019s Q2 2024 revenue growth?\u201d Your retriever has no idea this chunk is even about ACME Corp, let alone Q2 2024. The company name and the quarter were mentioned two paragraphs earlier, in a chunk that got split away during preprocessing. The embedding for this chunk carries none of that information, so it barely resembles the query, and it gets buried under chunks that are only superficially related.<\/p>\n This is not a rare edge case. It is the default behavior of naive chunking, and it is the single biggest reason RAG systems return \u201cI could not find relevant information\u201d or, worse, confidently hallucinate an\u00a0answer.<\/p>\n The fix Anthropic proposed is almost embarrassingly simple: before you embed or index a chunk, prepend a short piece of context that situates it within the full document. Instead of embedding<\/p>\n \u201cThe company\u2019s revenue grew by 3% over the previous quarter.\u201d<\/p>\n you embed<\/p>\n \u201cThis chunk is from an SEC filing on ACME Corp\u2019s Q2 2024 performance. The previous quarter\u2019s revenue was $314 million. The company\u2019s revenue grew by 3% over the previous quarter.\u201d<\/p>\n Now the chunk carries the company name, the time period, and the surrounding numbers, all inside the text that gets embedded and indexed. Both your vector search and your keyword search suddenly have something real to match\u00a0against.<\/p>\n The two components that make this work\u00a0are:<\/p>\n Contextual Embeddings<\/strong> Every chunk gets a short, chunk specific context written by an LLM before embedding. This context is generated using the whole document as reference, so it captures things like the document\u2019s subject, the section it belongs to, and any entities or dates that got separated from the chunk during splitting.<\/p>\n Contextual BM25<\/strong> The same contextualized chunk is also indexed for keyword search using BM25. This matters because embeddings alone still miss exact matches like order numbers, product codes, or specific dates, the same limitation we covered in the keyword search article in this series. Contextual BM25 gives you that exact match capability on top of contextualized text.<\/p>\n Run both together in a hybrid retriever, and you get chunks that are both semantically rich and precisely searchable.<\/p>\n You might be thinking, why not just make chunks bigger so they carry more context naturally(That was my first thought too \ud83e\udd14).A few reasons this does not hold\u00a0up:<\/p>\n Bigger chunks dilute relevance.<\/strong> A 2000 token chunk covering five different topics gets a muddy embedding that does not represent any single topic well. Precision drops even as recall might improve slightly.<\/p>\n Bigger chunks blow up your context window.<\/strong> If you retrieve the top 10 chunks and each one is huge, you are burning tokens on irrelevant surrounding text instead of giving the LLM more distinct, relevant pieces of information.<\/p>\n Contextual retrieval keeps chunks small and precise, and adds context as metadata baked into the text itself<\/strong>, rather than solving the problem by brute force. You get the recall benefits of bigger chunks without the\u00a0noise.<\/p>\n Here is a working pipeline you can adapt to your own documents.<\/p>\n from anthropic import Anthropic<\/p>\n client = Anthropic() If you are worried about cost, Claude\u2019s prompt caching handles this well. You cache the full document once and reuse that cached version across every chunk in that document, so you are only paying full price for the document tokens a single\u00a0time.<\/p>\n def build_contextualized_chunks(document_text, chunks): from rank_bm25 import BM25Okapi def build_hybrid_index(contextualized_chunks, embedding_model): def hybrid_retrieve(query, chunks, embeddings, bm25, embedding_model, top_k=20): For the final accuracy boost, add a reranker (Cohere\u2019s rerank model or a cross encoder) on top of these top 20 results before passing the final 3 to 5 chunks to your LLM. This is the step that took Anthropic\u2019s numbers from 49% failure reduction to\u00a067%.<\/p>\n Use a cheap, fast model for context generation. You do not need your most expensive model to write a two sentence summary of where a chunk sits in a document.Cache the full document when generating context for multiple chunks from the same source. This is where prompt caching saves real money at\u00a0scale.Keep the generated context short, 50 to 100 tokens is usually enough. Long context defeats the purpose of keeping chunks\u00a0small.Always pair contextual embeddings with contextual BM25. Using only one leaves accuracy on the\u00a0table.Add a reranking step if your use case can tolerate the extra latency. The accuracy gain is significant.Test on your own domain before trusting published numbers. Anthropic\u2019s benchmarks used codebases, fiction, ArXiv papers, and science papers, your documents may behave differently.Monitor your actual retrieval failure rate before and after, do not assume the improvement transfers without measuring it.<\/p>\n Contextual retrieval is one of those techniques that feels obvious once you see it, and that is usually a sign it is worth adopting. It does not require a new vector database, a new architecture, or months of engineering work. It requires rethinking one step in your pipeline: what actually gets embedded and\u00a0indexed.<\/p>\n If your RAG system has been giving vague or wrong answers and you have already tuned your chunk size, tried different embedding models, and added reranking, this is very likely the missing\u00a0piece.<\/p>\n Thank you for following this <\/strong>RAG tutorial series<\/strong><\/a>!<\/strong> If you found this helpful, please give it a clap \ud83d\udc4f and share with others building RAG\u00a0systems.<\/p>\n Contextual Retrieval: Anthropic\u2019s Method for Cutting RAG Failures<\/a> was originally published in Coinmonks<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>","protected":false},"excerpt":{"rendered":" Introduction Welcome back!!! If you have been following my RAG series, you already know that chunking is where most retrieval pipelines quietly break. You split a document into neat little pieces, embed each one, store it in a vector database, and assume the system now understands your content. It does not. Each chunk sits alone, […]<\/p>\n","protected":false},"author":0,"featured_media":197241,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-197240","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-interesting"],"_links":{"self":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts\/197240"}],"collection":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=197240"}],"version-history":[{"count":0,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/posts\/197240\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=\/wp\/v2\/media\/197241"}],"wp:attachment":[{"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=197240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=197240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mycryptomania.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=197240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The Real Problem With\u00a0Chunking<\/h3>\n
What Contextual Retrieval Actually\u00a0Does<\/h3>\n
Why This Works Better Than Alternatives<\/h3>\n
Implementation, Step by\u00a0Step<\/h3>\n
Step 1: Generate Context for Each\u00a0Chunk<\/h3>\n
CONTEXT_PROMPT = “””<document>
{full_document}
<\/document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk_content}
<\/chunk>
Please give a short, succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.”””
def generate_chunk_context(full_document, chunk_content):
response = client.messages.create(
model=”claude-opus-4-8″,
max_tokens=150,
messages=[{
“role”: “user”,
“content”: CONTEXT_PROMPT.format(
full_document=full_document,
chunk_content=chunk_content
)
}]
)
return response.content[0].text<\/p>\nStep 2: Build the Contextualized Chunks<\/h3>\n
contextualized_chunks = []
for chunk in chunks:
context = generate_chunk_context(document_text, chunk)
contextualized_chunk = f”{context}nn{chunk}”
contextualized_chunks.append(contextualized_chunk)
return contextualized_chunks<\/p>\nStep 3: Index for Both Embedding Search and\u00a0BM25<\/h3>\n
import numpy as np<\/p>\n
# Embedding index
embeddings = embedding_model.encode(contextualized_chunks)
# BM25 index
tokenized = [chunk.lower().split() for chunk in contextualized_chunks]
bm25 = BM25Okapi(tokenized)
return embeddings, bm25<\/p>\nStep 4: Retrieve With Both, Then\u00a0Rerank<\/h3>\n
# Semantic search
query_embedding = embedding_model.encode([query])[0]
semantic_scores = np.dot(embeddings, query_embedding)
# Keyword search
tokenized_query = query.lower().split()
bm25_scores = bm25.get_scores(tokenized_query)
# Combine, normalize both to 0-1 range first
semantic_norm = semantic_scores \/ (semantic_scores.max() + 1e-8)
bm25_norm = bm25_scores \/ (bm25_scores.max() + 1e-8)
combined = 0.5 * semantic_norm + 0.5 * bm25_norm
top_indices = combined.argsort()[-top_k:][::-1]
return [chunks[i] for i in top_indices]<\/p>\nBest Practices for Contextual Retrieval<\/h3>\n
Conclusion<\/h3>\n