Your RAG Pipeline Is Lying to You: How to Really Measure Retrieval Quality

Every RAG vendor claims 95%+ accuracy. The number is almost always misleading. Here's why: most RAG evaluations measure retrieval recall (did we find the right document?) not faithfulness (did the answer actually match what the document says?).

The three metrics that matter

Retrieval recall is necessary but not sufficient. A RAG pipeline can retrieve the right document and still produce a hallucinated answer. We measure three things:

• Citation accuracy: Does every claim in the answer trace back to a specific chunk in the retrieved documents? If the answer says "Revenue grew 23%" but the source document says 17%, that's a hallucination — even if the right document was retrieved.
• Multi-hop reasoning: Can the pipeline answer questions that require synthesizing information across 2+ documents? "Compare Q3 revenue to Q2 and explain the difference" requires finding both quarters, computing the delta, and explaining it.
• Negative coverage: Can the pipeline correctly say "I don't know" when the answer isn't in the documents? Most RAG pipelines hallucinate rather than admitting ignorance.

How our RAG API works

Our RAG endpoint at mazzagrp.com is built around these three metrics:

curl -X POST https://api.mazzagrp.com/v1/rag-query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What was our Q3 2024 revenue?",
    "documents": ["q3-report.pdf", "q3-earnings.pdf"],
    "require_citations": true
  }'

The response includes the answer, inline citations (chunk IDs + page numbers), and a confidence score. If confidence is below threshold, it returns "I cannot find this information in the provided documents" instead of guessing.

Chunk strategy matters more than model choice

We tested 5 chunking strategies (fixed-size, sentence, paragraph, semantic, and hybrid) across 1,000 enterprise documents. The results:

• Fixed-size (500 tokens): 62% citation accuracy
• Sentence-level: 71% accuracy, but high latency (too many chunks)
• Paragraph-level: 78% accuracy, good balance
• Semantic chunking: 84% accuracy — the winner
• Hybrid (semantic + overlap): 89% accuracy — what we use

The chunking strategy improved accuracy more than switching from GPT-4 to Claude. Context engineering beats model selection.

Try it yourself

Our RAG API costs $0.20 per query (USDC on Base, no subscription). Upload documents via the /v1/doc/extract endpoint first, then query with /v1/rag-query. Every answer comes with citations you can verify.