Strong RAG interviews focus on diagnosis and trade-offs. Can you detect where quality dropped, explain why, and propose fixes with clear cost implications? This guide gives you 25 high-signal scenarios and concise answer directions.
Retrieval Quality Scenarios
1. Good docs, wrong chunks retrieved
Likely causes: poor chunk boundaries, embedding mismatch, missing query rewrite. Fix by re-chunking semantically and adding hybrid retrieval.
2. Exact error codes never found
Dense retrieval misses lexical precision. Add sparse search (BM25) and blend with dense scores.
3. Answers improve when k is very high
Recall problem. Improve first-stage retrieval and reranking so you can keep smaller k and lower latency.
4. Old policy appears after policy update
Index freshness issue. Add document versioning and stale-index invalidation jobs.
5. Similar queries get inconsistent passages
Query normalization is weak. Add rewrite and canonicalization (acronyms, synonyms, product aliases).
Grounding and Hallucination Scenarios
6. Model gives fluent but uncited claims
Prompt does not enforce evidence mapping. Require citations per claim and reject uncited outputs.
7. Model answers despite no evidence
Missing abstain behavior. Add explicit fallback phrase and confidence threshold guard.
8. Conflicting sources produce overconfident answers
Need conflict handling rule: acknowledge disagreement and cite both sources.
9. Model ignores strongest chunk in middle position
Lost-in-the-middle effect. Reorder context so top evidence appears first and last.
10. Citations refer to irrelevant docs
Post-processing bug or hallucinated IDs. Validate citation IDs against retrieved set before response.
Latency and Cost Scenarios
11. P95 latency doubled after reranker launch
Reranker candidate set too large. Reduce candidate pool, cache embeddings, and use async retrieval paths.
12. Token bills spike after adding context
Context bloat. Compress chunks, deduplicate passages, and cap per-source chunk count.
13. Throughput collapses during traffic bursts
No admission control. Add queue limits, tiered model routing, and graceful fallbacks.
14. Multi-turn chat re-retrieves everything each turn
Conversation memory strategy missing. Cache relevant context windows by thread and invalidate smartly.
15. High-quality model is too expensive for all users
Route by complexity: smaller model for easy tasks, premium model for high-risk tasks.
Data and Security Scenarios
16. User sees another tenant's document
Critical filter failure. Enforce tenant metadata filters before retrieval and at response validation layer.
17. Deleted document still appears in answers
Soft deletes not propagated to index. Add delete tombstones and periodic consistency checks.
18. PII appears in generated summary
Add redaction during ingestion and policy checks before final output.
19. Non-English docs retrieve poorly
Embedding model mismatch for multilingual data. Switch to multilingual embeddings and language-aware retrieval.
20. Tables are ignored in answers
Ingestion pipeline strips structure. Use parsers that preserve table rows and headers as structured chunks.
Evaluation and Monitoring Scenarios
21. Offline metrics look good, users still unhappy
Benchmark not representative. Rebuild eval set from real production queries.
22. Retrieval Recall@k stable, answer quality drops
Generation stage regressed. Investigate prompt changes, context ordering, and model version updates.
23. Drift appears after product terminology changes
Index and rewrite dictionary outdated. Schedule domain vocabulary refreshes.
24. Weekly deploys cause unpredictable quality swings
No gated rollout. Add canary traffic and regression thresholds before full release.
25. Team cannot explain why a bad answer happened
Observability gap. Log query, retrieved IDs, prompt version, model, and output score for every request.
How to Use This for Interview Prep
- Practice a 60-second diagnosis for each scenario.
- Always mention one metric and one mitigation.
- Discuss trade-offs: quality, cost, latency, and risk.
Interviewers look for systems thinking. Show that you can trace failures across retrieval, generation, and operations.