How to build a RAG pipeline that doesn't hallucinate

Retrieval-augmented generation is easy to stand up and hard to trust. The first version — embed some docs, stuff the top matches into a prompt — works in a demo and then confidently invents answers the moment a user asks something the retrieval missed. Fixing that is mostly about the parts nobody screenshots.

Hallucination is usually a retrieval failure, not a model failure

When a RAG system makes things up, the model is often doing its job with bad inputs. If the right chunk never made it into context, the model fills the gap. So the highest-leverage work is upstream, in retrieval quality, not in prompt-tuning the generation step.

Chunking decides your ceiling

Chunk too large and you dilute the relevant sentence among noise; too small and you sever the context that makes it meaningful. Practical guidance:

• Chunk on semantic boundaries (headings, paragraphs), not a fixed character count that slices sentences in half.
• Keep a bit of overlap so a fact near a boundary isn’t orphaned.
• Store useful metadata (source, section, date) with each chunk — you’ll need it for filtering and for citations.

Retrieve better than pure vector search

Dense embeddings miss exact terms; keyword search misses paraphrase. Hybrid retrieval (combine vector similarity with keyword/BM25) plus a reranker on the top candidates is the single biggest quality jump most pipelines are missing. Retrieve broadly, then rerank down to the few chunks that actually answer the question.

Ground the generation and make refusal safe

• Instruct the model to answer only from the provided context and to say it doesn’t know when the context doesn’t cover the question. A system that can say “not in the docs” is more valuable than one that always answers.
• Require citations back to source chunks. This both helps users trust answers and gives you a way to spot when the model is drifting off-source.
• Return the sources in the UI. Visible citations change user behaviour and expose retrieval gaps fast.

You can’t improve what you don’t measure

Build an evaluation set of real questions with known good answers before you tune anything. Track two things separately:

• Retrieval quality: did the right chunk get retrieved at all? (If not, no prompt fixes it.)
• Answer quality: given the right context, was the answer correct and grounded?

Separating these tells you where to spend effort. Most teams discover their “hallucination problem” is a retrieval-recall problem, and a reranker plus better chunking fixes more than any model swap.

The short version

A RAG system that doesn’t hallucinate is one that retrieves the right context, admits when it can’t, and cites what it used. Get chunking and hybrid retrieval right, add a reranker, force grounded answers with citations, and measure retrieval and generation separately. Do that and RAG stops being a demo and starts being something you can put in front of users.