RAG Is a Retrieval Problem Wearing an LLM Costume

PDF-RAG-System (LangChain + ChromaDB + Mistral), AI Study Assistant, AI Research Assistant — three RAG systems, one recurring lesson: the model is rarely the weak link. Retrieval is. If the right chunk isn't in the context window, no amount of model quality saves you. The model can only be as good as the evidence you hand it.

Chunking is a design decision

Fixed-size chunks with no overlap chop tables, equations and reference lists in half. The retrieval layer then faithfully returns fragments of broken content, and the model gets blamed for hallucinating an answer the context never contained.

I now chunk along structure — headings, paragraphs, page boundaries — keep a small overlap between chunks, and store the source page with every chunk so an answer can point at where it came from. The best chunking heuristic is domain-specific: an academic PDF wants different boundaries than a policy handbook, and knowing which structure matters is actual product work.

Fix retrieval before blaming the model

In every one of these projects the cheapest quality win was on the retrieval side, in this order: better embeddings first, then a hybrid of vector and keyword search — vector search catches paraphrase, keyword search catches exact identifiers, model numbers and names that embeddings blur — and finally a reranking pass over the top candidates to fix the ordering the first-stage retrieval got wrong.

Swapping in a bigger LLM to cover up a wrong-context answer is the expensive way to hide a retrieval bug. The model will confidently write something plausible over the wrong evidence, which is strictly worse than a retrieval miss you can measure.

Measure retrieval directly

Once I started scoring retrieval on its own — does the gold chunk appear in the top-k? — debugging stopped being guesswork. If retrieval recall is high and answers are still wrong, the problem is the model or the prompt. If recall is low, no prompt engineering in the world helps. Two numbers, two different fixes, and the measurement costs an afternoon.

The contract that matters

A RAG system is only useful if it can say "I don't know". Every one of my systems returns citations with the answer and refuses when the retrieved context is thin — that single rule does more for trust than any prompt tweak I've tried. Users forgive an honest gap. They do not forgive an invented citation.

The loop I now always run

Build the smallest honest pipeline, measure retrieval recall and answer quality separately, fix retrieval first, and only then argue about models. It's boring, and it has never once failed to move quality faster than any other order of operations.