Customer Support RAG Project: Evaluation Plan

Answer First

An evaluation plan for a customer support RAG portfolio project needs three distinct measurement layers, because a single end-to-end “does the answer look right” check hides which component actually failed: retrieval evaluation (did the system find the right source document), generation evaluation (did the model use that source correctly), and end-to-end evaluation (did the final answer satisfy the user’s actual question). This page defines each metric, how to compute it, where the labeled dataset comes from, and the specific judge-calibration step that separates a defensible evaluation from a rubber-stamp one.

Scope and Assumptions

This page assumes the architecture described in the companion Customer Support RAG Project: Architecture page on this site (hybrid retrieval, citation enforcement, low-confidence fallback), and gives the evaluation plan that validates that system before and after changes. It assumes no existing production traffic to draw eval questions from, since this is a portfolio project rather than a live product — so it includes guidance on constructing a labeled set from scratch. It does not cover A/B testing infrastructure, since a portfolio project rarely has the traffic volume to run a statistically valid online experiment; the plan below is offline-evaluation-first with a lightweight online monitoring layer for when the project is deployed publicly.

Layer 1: Retrieval Evaluation

Metric: Recall@k. Of the questions in the labeled set, what fraction had the correct source chunk appear in the top-k retrieved results. This is computed independently of generation quality — it isolates whether the retrieval layer is doing its job before generation has a chance to compensate or compound the error.

Calculation: For each labeled question, mark the ground-truth chunk ID(s) that contain the answer. Run retrieval, check whether any ground-truth chunk ID appears in the top-k returned results. Recall@k = (questions where ground truth appeared in top-k) / (total questions).

Dataset construction: Since this is a portfolio project without production logs, build the labeled set manually: pick 50-80 realistic support questions covering the corpus’s major topics (aim for coverage across document types, not just the easiest topic), and for each, manually identify which chunk(s) in the ingested corpus actually contain the answer. This manual labeling step is the most time-consuming part of the evaluation plan and is also the part most candidates skip — doing it thoroughly is itself evidence of engineering rigor worth calling out in a project write-up.

Threshold source: There is no universal “good” recall@5 number independent of corpus difficulty; state your own measured baseline (e.g., naive fixed-size chunking, dense-only retrieval) and report relative improvement from hybrid search and re-ranking, rather than citing an external benchmark number as if it transfers to your specific corpus.

Layer 2: Generation Evaluation

Metric: Faithfulness (also called groundedness). Given the retrieved chunks the generator actually received, does the generated answer’s claims trace back to those chunks, or does it introduce information not present in the retrieved context (hallucination).

Calculation via LLM-as-judge: For each generated answer, prompt a separate judge model with the retrieved chunks and the generated answer, and ask it to classify each sentence in the answer as “supported by the provided context,” “contradicted by the provided context,” or “not present in the provided context” (this third category is the hallucination signal — the answer said something the retrieved chunks never stated). Faithfulness score = fraction of sentences marked “supported.”

Judge calibration — the step most projects skip: Before trusting the judge model’s classifications, validate it against a small human-labeled sample. Manually classify 20-30 generated answers yourself using the same three-way rubric, then run the judge model on the same 20-30 answers, and compute agreement (percent exact match, or Cohen’s kappa for a stricter measure). If agreement is below roughly 80%, the judge prompt needs revision — usually because the classification categories are underspecified or the judge is not given clear enough instructions on what counts as “contradicted” versus merely “not explicitly stated.” Reporting a faithfulness number without this calibration step is a common failure mode this site’s writing standards flag: uncalibrated LLM-judge numbers should not be presented as if they carry the same confidence as a human-labeled metric.

Dataset: Reuse the same 50-80 question set from Layer 1, since faithfulness evaluation needs the same questions run end-to-end through retrieval and generation, not a separate question set.

Layer 3: End-to-End Evaluation

Metric: Answer correctness / helpfulness rating. Does the final answer actually resolve the user’s question, on a simple rubric (correct, partially correct, incorrect, appropriately declined).

Calculation: Human-rate each of the 50-80 answers on this four-point rubric. This layer is intentionally human-rated rather than LLM-judged, because correctness in a support context often depends on nuance an automated judge is more likely to miss — for example, whether an answer that is technically accurate but omits a critical caveat (e.g., a refund policy exception) should count as “correct.” State in the project write-up that this layer is manually rated and why, since transparently choosing manual rating for the layer where it matters most is itself a signal of good evaluation judgment.

“Appropriately declined” as a first-class outcome: Given the low-confidence fallback in the architecture (returning “I don’t have enough information” rather than guessing), a correct decline should count as a pass, not a failure, when the ground truth question genuinely falls outside the corpus. Build 5-10 out-of-scope questions into the eval set specifically to test this — without them, the evaluation plan cannot verify the fallback behavior actually works.

Worked Example: A Completed Evaluation Run

Interpretation: The retrieval improvement (hybrid search + re-ranking) explains most of the downstream gain, since faithfulness and end-to-end correctness moved by roughly the same magnitude as recall — this is the kind of causal reasoning an evaluation plan should surface, not just a table of numbers with no interpretation attached.

Trade-offs Table: Evaluation Approach Choices

Decision Rubric: How Often to Re-run Each Layer

• Retrieval evaluation (Layer 1): re-run on every chunking or retrieval-logic change, since it is cheap to compute and directly measures the component being changed.
• Generation evaluation (Layer 2): re-run on every prompt template or generation-model change, and re-calibrate the judge any time the judge model itself is upgraded, since a new judge model version can shift its classification behavior even with an unchanged prompt.
• End-to-end evaluation (Layer 3): re-run before any deploy to a public-facing version of the project, and periodically (monthly, if the project stays live) even with no code changes, since document corpus drift (stale or updated source docs) can silently degrade correctness without any code change to blame.

Handling Evaluation Set Bias

A labeled evaluation set built by one person (typically the project author, for a portfolio project) carries a specific bias risk worth naming explicitly in the write-up: the author already knows the corpus well and tends to write questions phrased close to the document’s own language, which inflates retrieval recall relative to how real users actually phrase questions — real users use synonyms, misspellings, and indirect phrasing the document text does not contain verbatim. Mitigate this by deliberately writing a subset of eval questions (aim for at least a third) in paraphrased or colloquial language rather than terms lifted from the source documents, and by asking one other person, if available, to contribute a handful of questions independently, since a second author’s phrasing habits differ from the primary author’s in ways that surface gaps a single-author set would miss.

Regression Testing: Preventing Silent Evaluation Rot

An evaluation plan that is built once and never re-run stops being a safeguard the moment the underlying documents or model versions change. Treat the labeled eval set as a versioned artifact — store it in the project repository alongside the code, not as a one-off spreadsheet — and re-run all three layers automatically whenever the ingestion pipeline, retrieval logic, prompt template, or underlying model changes, ideally wired into a CI step that blocks a deploy if recall@5 or faithfulness regresses by more than a stated tolerance (for example, more than 5 percentage points versus the last approved run) without an explicit override. This turns the evaluation plan from a one-time report into an operational gate, which is the distinction between a project that demonstrates evaluation awareness and one that demonstrates evaluation as an ongoing discipline — interviewers at teams with mature ML operations practices specifically listen for this distinction.

Reporting the Results Honestly

When presenting this evaluation plan and its results in a portfolio write-up or interview, resist the temptation to report only the final, best-performing configuration. State the baseline you started from, the specific changes tested, and the delta each change produced, as in the worked example table above — this demonstrates the iterative, hypothesis-driven process rather than presenting a single polished number with no visible path to it. If a tested change did not improve results (for example, if adding a cross-encoder re-ranker produced a negligible faithfulness gain on your specific corpus, given the corpus was already small enough that dense retrieval alone reached near-ceiling recall), report that finding too — a null result reported honestly is stronger evidence of engineering judgment than a curated success-only narrative, and interviewers who ask “what didn’t work” are specifically checking for this kind of honesty.

Relevant Book Sample

The 0→1 AI Engineer Interview Playbook (ASIN B0H2CML9XD) includes a full chapter on presenting evaluation methodology in interviews, including how to answer follow-up probing on judge calibration specifically, which is one of the most common places candidates lose credibility when discussing LLM-as-judge evaluation. For the operational side of running this evaluation plan continuously after initial launch (drift detection, automated re-runs, alerting on metric regression), the companion title The 0→1 Loop Engineering Playbook covers that ground in more depth; its ASIN is not yet publicly verified, so confirm current availability before citing it.

Read the full evaluation methodology chapter: /go/B0H2CML9XD?source=ai-engineers-blog&page=aie-csrag-eval-001

Minimum Viable Evaluation Plan

If time only allows for a subset of this plan before a first deployment, the non-negotiable minimum is: a labeled set of at least 30 questions covering the corpus’s major topics, recall@5 measured against that set, and the out-of-scope decline test with at least 5 questions. Faithfulness scoring and full end-to-end human rating can follow in a second pass once the retrieval layer is validated, since a retrieval failure upstream makes generation-quality measurement premature — there is little value precisely measuring how well the model uses context it never should have received in the first place.

Sources and Freshness

The three-layer evaluation structure (retrieval / generation / end-to-end) and the judge-calibration methodology (human-agreement check before trusting an LLM judge) reflect standard, publicly documented RAG evaluation practice as of 2026. No comparative benchmark numbers from external published papers are cited as universal thresholds; all numeric thresholds in this page are presented as defensible starting points to validate against your own project’s data, per site policy against unverifiable comparative claims. Last verified: 2026-07. Next review: quarterly.

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.