Building RAG from scratch

I built a fifteen-stage RAG ladder and measured the first twelve stages on the same benchmark. Of ten recommended techniques, seven made retrieval worse, one was flat and two helped.

Read the guide Code

Made it worse

  • Chunking-0.010 to -0.020
  • RRF fusion-0.009 to -0.024
  • Reranking on dense-0.033
  • HyDE-0.048
  • Step-back prompting-0.066
  • Query decomposition-0.095
  • CRAG relevance gradingchance
  • Contextual retrieval-0.006

Helped

  • Weighted hybrid 0.7/0.3+0.010
  • Late interaction+0.011
  • Reranking on BM25+0.041

Change in nDCG@10 against a 0.759 baseline on BEIR SciFact, 5,183 abstracts and 300 labelled queries. The two that helped are the two where I measured the gap before applying the technique. None of these are bad techniques. They are conditional, and the condition is whether your pipeline is already good at the thing they fix. Reranking flipped sign inside this repo: -0.033 on a strong first stage, +0.041 on a weak one.

The fifteen levels

1Naive RAG2Measure first3Move to a real benchmark4Chunking5Keyword search (BM25)6Hybrid retrieval7Cross-encoder reranking8Query rewriting9Context enrichment10Multi-vector / late interaction11Grounded generation12Self-correction (CRAG, Self-RAG)13Agentic / multi-hop retrieval14GraphRAG15Production scale

More

Decision procedure, a build order with a diagnostic before each technique. Taxonomy of every RAG system by the problem it solves. Lab notebook, what broke and what I got wrong.