Almost every RAG tutorial stops at the part where it starts getting hard.

You load a PDF, chop it into 500-character pieces, embed them, drop them in a vector store, and pipe the top 5 results into an LLM. It works on the demo question. You ship it. Then a user asks something slightly off-distribution, the retriever returns three irrelevant chunks and two half-relevant ones, the model confidently hallucinates around the gaps, and now you're explaining to your stakeholders why the "AI search" gave a wrong answer about a refund policy.

The gap between a RAG demo and a RAG system is almost entirely in the retrieval layer and the evaluation loop — the two things tutorials gloss over. This guide builds the whole thing from primitives so you understand every moving part, then hardens each stage for production. By the end you'll have a hybrid retriever with reranking, a grounded generation step that knows when to say "I don't know," an evaluation harness that tells you whether any change actually helped, and a checklist of the production concerns that bite people six weeks after launch.

We'll write real Python. No framework magic — you can wire in LangChain or LlamaIndex later, but you'll know exactly what they're doing for you.

What RAG actually is (and when not to use it)

Retrieval-Augmented Generation is simple in concept: instead of relying on what a language model memorized during training, you retrieve relevant text at query time and inject it into the prompt so the model answers from that evidence. It buys you three things a bare LLM can't give you: knowledge that's fresh, knowledge that's private to your organization, and answers you can trace back to a source.

But RAG is not a default. Reach for it when your knowledge is large, changes often, or must be cited. If your knowledge fits in a single prompt, just put it in the prompt. If the task is reasoning-heavy rather than knowledge-heavy, retrieval won't help and may hurt. And if you need the model to deeply internalize a style or a skill rather than look up facts, that's a fine-tuning problem, not a retrieval one. The strongest production systems often combine all three; RAG is one tool, not the whole toolbox.

The two pipelines

Every RAG system is really two pipelines that meet at the vector store:

Most quality problems live in the online pipeline. Most cost and latency problems live in how you balance the two. Keep them mentally separate; they have different SLAs.

Part 1 — The 40-line RAG you should outgrow immediately

Let's build the naive version first, because you need to feel where it breaks. Install the essentials:

pip install sentence-transformers faiss-cpu anthropic
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic

embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env
MODEL = "claude-sonnet-4-5"     # use your provider's current model id

# 1. Naive fixed-size chunking
def naive_chunk(text, size=500):
    return [text[i:i + size] for i in range(0, len(text), size)]

# 2. Index
docs = open("knowledge.txt", encoding="utf-8").read()
chunks = naive_chunk(docs)
emb = embedder.encode(chunks, normalize_embeddings=True).astype("float32")
index = faiss.IndexFlatIP(emb.shape[1])   # inner product on normalized = cosine
index.add(emb)

# 3. Retrieve + generate
def ask(query, k=5):
    q = embedder.encode([query], normalize_embeddings=True).astype("float32")
    _, ids = index.search(q, k)
    context = "\n\n".join(chunks[i] for i in ids[0])
    msg = client.messages.create(
        model=MODEL, max_tokens=1024,
        messages=[{"role": "user",
                   "content": f"Context:\n{context}\n\nQuestion: {query}"}],
    )
    return msg.content[0].text

print(ask("What is our refund window?"))

This runs. It will even answer some questions correctly. Now here is everything wrong with it, which is the actual curriculum:

  1. Fixed-size chunking slices sentences in half and merges unrelated paragraphs, so embeddings represent garbage.
  2. Dense-only retrieval misses exact-keyword matches — product codes, names, error strings — that semantic similarity is bad at.
  3. No reranking, so the top-k is whatever the embedding model's first guess was, noise included.
  4. No query handling — a vague or multi-part question goes straight to the retriever unimproved.
  5. No grounding instructions, so the model blends retrieved text with its own training data and hallucinates in the seams.
  6. No "I don't know" path, so missing information becomes a confident wrong answer.
  7. No evaluation, so you have no idea if any change you make helps or hurts.

We'll fix all seven.

Part 2 — Chunking: the decision that quietly determines your ceiling

Chunking is the most under-respected stage in RAG. Your retriever can only return chunks you created, and your model can only reason over text inside them. Bad chunks cap your quality before retrieval even runs.

The core tension: small chunks retrieve precisely but lose context; large chunks carry context but dilute the embedding and waste tokens. You're tuning that trade-off.

Skip fixed-size. The cheapest real upgrade is sentence-aware chunking with overlap — never split mid-sentence, and let consecutive chunks share a little context so an idea straddling a boundary survives:

import re

def sentence_chunk(text, max_chars=800, overlap_sentences=2):
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    chunks, current, length = [], [], 0
    for s in sentences:
        if length + len(s) > max_chars and current:
            chunks.append(" ".join(current))
            current = current[-overlap_sentences:]      # carry overlap forward
            length = sum(len(x) for x in current)
        current.append(s)
        length += len(s)
    if current:
        chunks.append(" ".join(current))
    return chunks

Better still is structure-aware chunking: split on the document's own boundaries — Markdown headings, HTML sections, slide titles, table rows — and attach that structure as metadata. A chunk that knows it came from "Section 4.2 — Refund Policy" of "Terms v3" is dramatically more retrievable and lets you show users a real citation. For production, langchain_text_splitters.RecursiveCharacterTextSplitter and unstructured handle messy real-world files (PDFs with columns, tables, scanned pages) far better than anything you'll hand-write.

The frontier option is semantic chunking: embed sentences, then start a new chunk wherever adjacent-sentence similarity drops below a threshold, so splits land on genuine topic shifts. It's more expensive to compute and not always worth it — measure before you adopt it.

Practical defaults to start from, then tune against your eval set: 600–1,000 characters (roughly 150–250 tokens) per chunk, 10–20% overlap, and always carry metadata — source, section, page, last_updated. Measure chunk size in tokens (use tiktoken or your model's tokenizer), not characters, once you start optimizing context budgets.

Represent each chunk as a small dict so metadata travels with it everywhere:

def make_chunks(raw_text, source):
    return [
        {"id": f"{source}::{i}", "text": t, "metadata": {"source": source}}
        for i, t in enumerate(sentence_chunk(raw_text))
    ]

Part 3 — Embeddings and the vector store

The embedding model converts text into vectors where "close" means "semantically similar." Your choices have real consequences, so decide deliberately rather than defaulting to whatever the first tutorial used.

Picking an embedding model. The trade-offs are dimension (storage and search cost), domain fit, and whether you host it. A small open model like bge-small-en-v1.5 (384-dim) is fast, free, and runs locally — perfect to start. Larger open models (bge-large, e5-large) and hosted APIs (Voyage, Cohere, OpenAI's text-embedding-3) trade cost for accuracy, especially on domain-specific or multilingual text. Two rules that save pain later: embed queries and documents with the same model, and respect the model's instructions — BGE models, for example, want a prefix on the query side only:

QUERY_PREFIX = "Represent this sentence for searching relevant passages: "

Picking a vector store. This is mostly an operational decision, not a quality one:

Start with FAISS or pgvector. Don't reach for a managed vector DB until scale or ops actually demand it; an early dependency you don't need is just risk.

Part 4 — Retrieval: where quality is won or lost

Here's the single highest-leverage upgrade in this entire guide. Naive RAG does dense-only retrieval, which is semantically smart but keyword-blind. Ask it about error code E_4012 or a part number and cosine similarity shrugs. The fix is hybrid retrieval: run dense (embeddings) and sparse (BM25, classic keyword scoring) in parallel and fuse the results.

The clean way to fuse two ranked lists that live on different score scales is Reciprocal Rank Fusion — it ignores the raw scores and only trusts the ranks, which makes it robust and tuning-free.

Then add a reranker. Your first-stage retrieval optimizes for recall — cast a wide net, grab 20–40 candidates. A cross-encoder reranker reads the query and each candidate together (not as separate vectors) and scores true relevance far more accurately, so you can shrink 40 noisy candidates down to the 5 clean ones the model actually sees. This one step routinely moves answer quality more than any prompt tweak.

Here's the whole retrieval stack as one class:

pip install rank-bm25
import faiss
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder

class HybridRetriever:
    def __init__(self,
                 embed_model="BAAI/bge-small-en-v1.5",
                 reranker_model="BAAI/bge-reranker-base"):
        self.embedder = SentenceTransformer(embed_model)
        self.reranker = CrossEncoder(reranker_model)
        self.chunks, self.index, self.bm25 = [], None, None
        self.query_prefix = "Represent this sentence for searching relevant passages: "

    def build(self, chunks):
        self.chunks = chunks
        texts = [c["text"] for c in chunks]
        emb = self.embedder.encode(
            texts, normalize_embeddings=True, show_progress_bar=True
        ).astype("float32")
        self.index = faiss.IndexFlatIP(emb.shape[1])
        self.index.add(emb)
        self.bm25 = BM25Okapi([t.lower().split() for t in texts])

    def _dense(self, query, k):
        q = self.embedder.encode(
            [self.query_prefix + query], normalize_embeddings=True
        ).astype("float32")
        _, ids = self.index.search(q, k)
        return list(ids[0])

    def _sparse(self, query, k):
        scores = self.bm25.get_scores(query.lower().split())
        return list(np.argsort(scores)[::-1][:k])

    @staticmethod
    def _rrf(rankings, k=60):
        fused = {}
        for ranking in rankings:
            for rank, doc_id in enumerate(ranking):
                fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
        return [doc_id for doc_id, _ in sorted(fused.items(), key=lambda x: -x[1])]

    def retrieve(self, query, k_dense=20, k_sparse=20, k_final=5):
        dense = self._dense(query, k_dense)
        sparse = self._sparse(query, k_sparse)
        candidates = self._rrf([dense, sparse])[: max(k_dense, k_sparse)]
        pairs = [[query, self.chunks[i]["text"]] for i in candidates]
        scores = self.reranker.predict(pairs)
        ranked = [i for _, i in sorted(zip(scores, candidates), key=lambda x: -x[0])]
        return [self.chunks[i] for i in ranked[:k_final]]

Query transformation: fix the question before you search it

Users don't write retrieval-friendly queries. They're vague ("does it cover this?"), multi-part ("compare the basic and pro refund terms"), or full of pronouns from earlier in a conversation. Three techniques, cheapest first:

def rewrite_query(raw_query, client, model):
    msg = client.messages.create(
        model=model, max_tokens=128,
        system="Rewrite the user's question as a single, standalone search query. "
               "Resolve pronouns, keep all key terms, output only the query.",
        messages=[{"role": "user", "content": raw_query}],
    )
    return msg.content[0].text.strip()

Metadata filtering is non-negotiable in production

The moment you have multiple users, tenants, or document versions, semantic similarity alone is dangerous — it will happily retrieve another customer's document because it's topically similar. Filter on metadata before (or during) the vector search: restrict to the user's allowed sources, the current document version, the right language. This is a security control, not just a quality one. Most production vector stores let you pass a filter alongside the query; with raw FAISS you maintain the filter yourself by pre-masking candidate ids.

Part 5 — Generation: grounding, citations, and the courage to say "I don't know"

Retrieval found the evidence. Generation has one job: answer from that evidence and nothing else, cite it, and refuse when the evidence isn't there. The naive version failed because it gave the model context but no instructions, so the model treated retrieval as a suggestion and filled gaps from training data.

Three things fix this: a system prompt that hard-constrains the model to the context, source ids it can cite, and an explicit escape hatch for missing information.

SYSTEM = """You are a precise assistant. Follow these rules without exception:
- Answer ONLY using the provided context. Do not use outside knowledge.
- After each claim, cite the source id in brackets, like [2].
- If the context does not contain the answer, reply exactly:
  "I don't have enough information to answer that based on the available documents."
- If sources conflict, say so and present both."""

def build_context(chunks):
    return "\n\n".join(
        f"[{i}] (source: {c['metadata'].get('source', '?')})\n{c['text']}"
        for i, c in enumerate(chunks)
    )

def answer(query, retriever, client, model):
    chunks = retriever.retrieve(query)
    context = build_context(chunks)
    msg = client.messages.create(
        model=model, max_tokens=1024, system=SYSTEM,
        messages=[{"role": "user",
                   "content": f"Context:\n{context}\n\nQuestion: {query}"}],
    )
    return msg.content[0].text, chunks

Two judgment calls worth making deliberately. Context ordering matters — many models attend more strongly to the start and end of a long context than the middle ("lost in the middle"), so put your highest-reranked chunks at the edges, not buried. And resist stuffing more chunks in. More context is not more accuracy; it's more distraction, more cost, and more latency. A tight 3–5 well-reranked chunks usually beats 15 mediocre ones.

Part 6 — Evaluation: the part that separates engineering from vibes

If you take one thing from this guide, take this: you cannot improve a RAG system you can't measure. Every change above — chunk size, hybrid weights, reranker on or off, HyDE or not — is a hypothesis. Without evaluation you're tuning by feel, and feel is wrong about half the time.

Evaluate the two pipelines separately, because they fail for different reasons. A wrong answer is either a retrieval failure (the evidence never showed up) or a generation failure (the evidence was there and the model still blew it). Conflating them sends you optimizing the wrong stage.

Step one: build a small evaluation set. You need 30–50 questions, each labeled with the chunk id(s) that actually answer it and a reference answer. Yes, this is manual work. It is the highest-ROI hour you will spend on the project. Pull real questions from logs or support tickets where you can.

Retrieval metrics — did the right chunk make it into the results?

def recall_at_k(retriever, dataset, k=5):
    hits = sum(
        bool({c["id"] for c in retriever.retrieve(ex["question"], k_final=k)}
             & set(ex["relevant_ids"]))
        for ex in dataset
    )
    return hits / len(dataset)

def mrr(retriever, dataset, k=10):
    total = 0.0
    for ex in dataset:
        retrieved = [c["id"] for c in retriever.retrieve(ex["question"], k_final=k)]
        for rank, cid in enumerate(retrieved, start=1):
            if cid in set(ex["relevant_ids"]):
                total += 1.0 / rank
                break
    return total / len(dataset)

Recall@k asks "is the answer in there at all?" — your retrieval ceiling. MRR asks "how high up?" — which matters because chunks near the top get read more carefully. Track both. If recall@k is low, fix chunking and retrieval. If recall is high but MRR is low, you need (better) reranking.

Generation metrics — given good evidence, was the answer good? The two that matter most are faithfulness (every claim is supported by the retrieved context — i.e., no hallucination) and answer relevance (it actually addresses the question). These are hard to score with string matching, so the standard approach is LLM-as-Judge: a second model grades the answer against the context.

JUDGE = """Given CONTEXT and an ANSWER, is every factual claim in the ANSWER
directly supported by the CONTEXT? Respond ONLY with JSON:
{"faithful": true or false, "unsupported_claims": ["..."]}"""

def faithfulness(answer_text, chunks, client, model):
    context = "\n\n".join(c["text"] for c in chunks)
    msg = client.messages.create(
        model=model, max_tokens=512, system=JUDGE,
        messages=[{"role": "user",
                   "content": f"CONTEXT:\n{context}\n\nANSWER:\n{answer_text}"}],
    )
    return msg.content[0].text  # parse the JSON downstream

LLM-as-Judge is powerful but not free of bias — judges drift, favor verbose answers, and prefer the first option in a pair. Calibrate the judge against a handful of human labels before you trust it, and re-check periodically. (This is a deep topic in its own right; the off-the-shelf library RAGAS packages faithfulness, answer relevance, context precision, and context recall if you'd rather not hand-roll them.)

Wire these into CI: every change to chunking, retrieval, or the prompt runs the eval set and you compare the numbers. That loop — change, measure, keep or revert — is the entire difference between a RAG system that improves over time and one that mysteriously degrades.

Part 7 — Production hardening: what bites you at week six

The system works on your laptop. Here's what production adds.

Cost and latency. Reranking and LLM calls dominate both. Practical levers: cache embeddings so you never re-embed unchanged chunks; add a semantic response cache so near-duplicate questions skip retrieval and generation entirely; rerank only the top 20–40 candidates, not everything; batch embedding jobs; and run dense, sparse, and metadata lookups concurrently rather than in series. Pick the smallest embedding and reranker models that pass your eval bar — bigger is not free.

Freshness and re-indexing. Documents change. You need incremental indexing (re-embed only what changed, keyed by a content hash), and you need to decide what happens to in-flight answers when an index updates. Version your index alongside your chunking config and embedding model — the day you switch embedding models, every vector must be recomputed, and a half-migrated index silently returns nonsense.

Monitoring. Log every query, the retrieved chunk ids and their scores, the final answer, and (critically) user feedback signals — thumbs, follow-up rephrasings, escalations to a human. Watch the "I don't have enough information" rate: a sudden rise usually means your index broke or drifted from the questions people now ask. Retrieval scores trending down over time is an early warning that your content and your traffic have diverged.

Security and safety. RAG opens an attack surface most teams miss: indirect prompt injection. Retrieved documents are untrusted input — a malicious instruction hidden in a PDF ("ignore previous instructions and...") becomes part of your prompt. Treat retrieved text as data, never as instructions: keep it clearly delimited, instruct the model to ignore directives found inside context, and don't let retrieved content trigger tool calls without a gate. Add PII handling (redact at ingestion or enforce access at query time) and per-user access control via metadata filters — never rely on the LLM to "remember" not to reveal something it was handed. In regulated domains (finance, healthcare, legal), keep an audit trail: which sources produced which answer, so any output can be defended later.

The pitfalls checklist

Before you ship, walk this list — these are the failures that show up again and again:

Where frameworks fit

Everything above is deliberately framework-free so you understand the mechanics. In practice, LangChain, LlamaIndex, and Haystack implement these primitives well — text splitters, retrievers, RRF, reranker integrations, eval hooks — and you should absolutely use them once you know what they're doing. The failure mode is reaching for a framework first, treating retrieval as a black box, and having no idea which knob to turn when quality is bad. Build it once by hand; then let the framework save you the boilerplate.

Conclusion

A production RAG system isn't a clever prompt — it's a retrieval engine with an LLM bolted on the end, and the engineering lives in the retrieval and the evaluation loop. Start with the naive version to feel the failures, then earn each upgrade with a measurement: structure-aware chunking, hybrid retrieval, a reranker, query rewriting, grounded generation that can decline, and an eval harness that tells you the truth. Do that, and you'll have something that survives contact with real users instead of dying on the second question.

The next post in this series goes one level deeper on the evaluation half — building a proper eval set, calibrating an LLM judge you can trust, and wiring it all into CI so your RAG system improves with every change instead of silently rotting.

Want the structured version?

The AI Engineering and AI Evals Pro tracks at PathtoAI Academy cover RAG, retrieval, and evaluation end to end with hands-on projects.