Practice RAG Retrieval Metrics Offline — A Tiny Stdlib Eval Loop (Synthetic Data)
Learning RAG raises a basic question: “I can chat with my docs, but is retrieval improving?”
This tiny offline evaluation loop uses synthetic documents, QA gold labels, and a toy lexical retriever. It is Python stdlib-first: no weights, API keys, vector database, or LLM judge.
Honest scope: a learning harness, not a production RAG stack. Synthetic scores are not business scores.
Retrieval metrics
RAG has retrieval (right documents?) and generation (faithful answer?). Start with:
| Metric | Meaning |
|---|---|
| Precision@k | Fraction of top-k items that are gold |
| Recall@k | Fraction of gold items found in top-k |
| Hit@k | Whether top-k contains a gold item |
If Hit@k is low, the generator had no fair chance. Faithfulness is N/A here because there is no generator; do not invent a score.
Minimal loop
docs.jsonl -> toy retriever -> top-k ids
qa.jsonl -> gold ids -> P@k / R@k / Hit@k
Tokenize question and docs, score word overlap, return top-k ids, and compare with gold ids. Use desensitized files only; never share names, internal URLs, or secrets.
With k=3 and one gold document, Precision@3 may be 0.33 even when you hit; Recall@3 can be 1.0 while ranking is noisy. High synthetic scores are not production readiness. Keep dataset and k fixed while changing one variable.
Try none, fixed, and sliding chunks. A toy rerank toggle is illustrative, not a production cross-encoder. With fewer than 30 questions, ignore a 0.02 swing.
Packs (optional)
Free Sample ($0): https://payhip.com/b/kdH7J Lite ($2.99): https://payhip.com/b/cmjJk Field Card ($4.99): https://payhip.com/b/Lrw6Y Pro ($11.99): https://payhip.com/b/UdQ0M
Honest limits: synthetic / educational only; toy lexical retriever; not production; scores are practice, not proof.