RAG Explained: Retrieval-Augmented Generation from Scratch

Retrieval-Augmented Generation, or RAG, combines two systems: information retrieval and language generation. Instead of asking an LLM to answer from memory, you first retrieve relevant documents and pass them to the model as context.

Why RAG Exists

Pure LLM apps fail in predictable ways: stale knowledge, hallucinated facts, and no citation path. RAG reduces these failures by grounding responses in data you control.

• Freshness: update your index without retraining the model.
• Traceability: attach source citations to every answer.
• Domain control: answers prioritize your documentation over public web noise.

The End-to-End Pipeline

1. Ingest: collect PDFs, markdown, support docs, tickets.
2. Chunk: split text into semantically coherent units.
3. Embed: convert chunks into vectors.
4. Index: store vectors and metadata in a searchable store.
5. Retrieve: fetch candidate chunks for a query.
6. Rerank: reorder candidates using a stronger relevance model.
7. Generate: answer from retrieved context with strict grounding rules.
8. Evaluate: measure retrieval and answer quality continuously.

Chunking Is Your Hidden Superpower

Most RAG quality problems start in chunking. Tiny chunks lose meaning. Huge chunks dilute relevance. Good defaults are:

• 150 to 350 tokens per chunk
• 10 to 20 percent overlap
• Metadata: source, section, timestamp, tenant id

If your data has structure, chunk by headings and sections instead of fixed character windows.

Retrieval Design Choices

Dense retrieval using embeddings is excellent for semantic similarity but can miss exact keyword matches. Sparse retrieval (BM25) does the opposite. Hybrid search combines both and usually wins in production.

final_score = alpha * dense_score + (1 - alpha) * sparse_score

Then apply reranking to top candidates. This step alone can dramatically improve answer precision.

Grounded Generation Prompt

Use strict instructions so the model does not improvise beyond the supplied evidence.

You are a grounded assistant.
Use only the provided context.
Cite source IDs for each factual claim.
If evidence is missing, respond: "Insufficient evidence in provided documents."

Minimal Python Skeleton

def rag_answer(query, retriever, llm):
    docs = retriever.search(query, k=6)
    context = "\n\n".join([d.text for d in docs])
    prompt = f"Context:\n{context}\n\nQuestion: {query}"
    return llm.generate(prompt)

Common Failure Modes

• Irrelevant retrieval: bad chunking or weak embeddings.
• Confident hallucinations: missing grounding constraints.
• Latency spikes: too many retrieved chunks or slow reranker.
• Cross-tenant leakage: missing metadata filters.

How to Evaluate RAG Properly

Track retrieval and generation separately:

• Retrieval: Recall@k, MRR, nDCG
• Generation: factuality, citation correctness, task success rate

Use a fixed benchmark dataset and run it on every pipeline change.

When Not to Use RAG

RAG is not always the right answer. If tasks are purely procedural and do not need external knowledge, standard workflows or fine-tuning may be simpler and cheaper.

RAG works best when knowledge changes faster than model training cycles.