RAG Evaluation
By Ai Flowi · Published
RAG evaluation checks whether retrieval finds the needed evidence and whether the generated answer uses that evidence correctly. Evaluate passage coverage, answer grounding and usefulness separately: an accurate-looking answer can conceal poor retrieval, while relevant passages can still produce an unsupported reply. Start with labelled questions and source passages before adding an automated evaluator.
This procedure uses AI Flowi Workflow catalogue components to organise a proposed evaluation run. They are documented in the catalogue; not production-tested by Ai Flowi. Evaluator outputs described here are interface contracts, not measured quality results or benchmark scores.
1. Build a question corpus with evidence labels
Keep each case in a CSV with a stable case identifier, question, answerability label, approved source version, required passage identifiers and a short acceptance note. Write labels from the documents, not from the chatbot’s current answer. Include questions that need an exception as well as a main rule; otherwise a system that drops exceptions can appear acceptable.
Follow the Load CSV interface and Data → DataFrame conversion reference for importing the labelled corpus.
A case identifier must describe the question and source version together. If a policy changes, preserve the old case record and create a new version of its evidence labels. Otherwise a changed expected answer can look like a model regression even when the model correctly followed the new policy. Keep cases with failed retrieval requests in the dataset; silently dropping them makes a comparison measure only the replies that happened to arrive.
| Case category | Gold evidence label | Failure exposed |
|---|---|---|
| Covered rule and exception | Both passage identifiers are required. | Retrieval or generation drops the exception. |
| Unanswerable request | No approved supporting passage exists. | An answer is manufactured from unrelated text. |
| Version conflict | Two sources disagree and require review. | A contradiction is concealed in a fluent answer. |
| Relevant-looking distractor | Only the passage about the requested subject qualifies. | Surface wording outranks applicable evidence. |
2. Capture the actual retrieval and answer inputs
Run the question against Knowledge Base.search_query with a fixed knowledge_base and top_k; retain metadata using include_metadata. Save Results beside the case identifier. Base creation and chunk inspection belong in the existing retrieval guide. For an answer path, use the separate grounded chatbot design and capture the text actually sent to the model.
A source passage in the database is not the same as a source passage in the prompt. Record the retrieved rows, the formatted context and the final answer. If formatting truncates a required exception, label that as a context-construction failure. Do not blame the generator for information that never reached its input, or count a claim as grounded because its source exists somewhere else.
3. Configure evaluation inputs per case
Per-case evaluation setup: example configuration, not a recorded test
| Component | Catalogue inputs to set | Inspection before continuing |
|---|---|---|
| Load CSV and Data → DataFrame | csv_file: reviewed corpus; then data_list: JSON List. | Confirm one identifiable case per resulting row. |
| Batch Run | df: prepared prompt table; column_name: reviewed prompt column; output_column_name: candidate_answer; model: configured LanguageModel. | This processes a column; it is not evidence that retrieval runs automatically for each row. |
| Cleanlab RAG Evaluator | query: case question; context: actual supplied evidence; response: saved answer; model: supported configured choice; quality_preset: "base"; api_key: your credential. | Review trust_score and explanations alongside source labels. |
| LangWatch Evaluator | evaluator_name: a configured provider evaluator; input: question; output: answer; contexts: evidence; expected_output: reviewed acceptance text. | The export supplies no evaluator-name choices; configure the provider before a trial. |
You need your own model and evaluator accounts, any required API keys and permission to transmit the synthetic corpus to those services. Providers bill separately. The catalogue gives typed evaluation inputs but no complete retrieval-to-batch orchestration recipe. Begin with one case, map its observed values explicitly, and verify each boundary before attempting a dataset run.
4. How do I keep coverage, grounding and usefulness distinct in RAG evaluation?
For retrieval coverage, compare returned passage identifiers with the gold identifiers. For grounding, inspect whether each material answer claim is supported by the context actually supplied. For usefulness, check whether the answer addresses the question and preserves applicable conditions. These checks can disagree: abstaining is appropriate on an unanswerable case but insufficient on a covered case.
Cleanlab RAG Evaluator exports switches named run_context_sufficiency, run_query_ease, run_response_groundedness and run_response_helpfulness. Enable the checks relevant to your rubric and retain other_scores, trust_explanation and evaluation_summary for inspection. The export does not define the scores’ scale, calibration or universal pass thresholds; a numeric output is not an authoritative truth label.
Cleanlab Evaluator has prompt and response inputs for general response evaluation. It lacks the separately named context and query contract of the RAG evaluator. Choose the interface that matches the evidence you want assessed, and retain human passage checks even when an automated explanation sounds persuasive.
5. Compare versions by case identifier
Freeze the corpus, source versions and evaluation rubric. Change one retrieval or prompt choice, then compare old and new outcomes for the same cases. In Table Operations, set operation to the exported option "Merge", merge_on_column to your case identifier and merge_how to "outer". Verify identifier uniqueness before joining the results.
For a worked comparison, imagine one case has a baseline answer but no candidate answer because the new retrieval request failed. An outer comparison should keep that case visible for failure review. An inner comparison would omit it and could make the candidate look cleaner. This is an expected comparison property, not a recorded Table Operations result; inspect the actual joined rows before summarising changes.
Save model selection, retrieval count, prompt text and evaluator settings with each run. A changed evaluator can alter the judgement even when answers are unchanged. Review cases that improved and cases that regressed, plus failed requests and missing outputs. Do not average away unanswerable-case failures or treat an evaluator timeout as a successful answer.
6. Use synthetic cases as candidates, not gold truth
aGenerate can draft mock rows from schema, instructions and an optional source table. Ask it for alternative question wording and distractors, then review them against approved passages before admitting them to the corpus. Keep generated candidates distinguishable from manually labelled cases. A model-created answer key can repeat the same misunderstanding as the system under test.
The platform evaluation checklist covers choosing and testing a builder; this guide evaluates a particular retrieval-and-answer task. Agent tool choices need their own observations, as explained in AI agent evaluation. Do not turn a response score into a claim about permissions, tool correctness or deployment reliability.
7. Test fallback handling after evaluation
Cleanlab Remediator exposes score, threshold, fallback_text and show_untrustworthy_response. Use a threshold only after understanding the chosen evaluator’s output and reviewing labelled cases. Its required score input accepts number; verify compatibility with the actual numeric output. A fallback rule does not repair missing passages or make an incorrect answer correct.
Do not use this proposed harness as an autonomous release gate for sensitive decisions. Review errors, source coverage and evaluator disagreements before changing the user-facing flow. Preserve prior outputs so a fallback can be audited without exposing an untrusted answer to users.
Evaluation budget
Plans: Go from USD8/month billed yearly (USD10 monthly), Plus from USD16 (USD20 monthly), Pro 100 from USD80 (USD100 monthly), Pro 200 from USD160 (USD200 monthly) and Pro 500 from USD400 (USD500 monthly), with 5,000 to 250,000 Flowi Tokens/month and top-ups; 1 token per node run; third-party API and model usage is not included.
Dataset rows, reruns and evaluator calls add usage independently of a production chatbot conversation. Estimate them from the observed run path rather than the number of cases alone.