Customer Support RAG Project: Architecture

Answer First

A portfolio-worthy customer support RAG project needs five components wired together with explicit interfaces: an ingestion pipeline that chunks and embeds support documents, a retrieval layer with hybrid search (not vector-only), a generation layer with citation enforcement, a feedback loop that captures thumbs-up/down signal, and an evaluation harness that runs before every deploy. This page gives the full architecture diagram in text form, the component-by-component build order, and the specific technical decisions that make this project read as production-grade rather than a notebook demo when you present it in an interview.

Scope and Assumptions

This is a project architecture page for engineers building a portfolio project to demonstrate RAG competence for job applications, not a page for teams shipping an internal enterprise support bot with existing infrastructure. It assumes access to a public or synthetic support-document corpus (product docs, FAQ pages, or a public dataset like the Twitter customer support dataset) and standard cloud infrastructure (a vector database, a hosted LLM API, basic compute for the ingestion pipeline). It assumes the reader can write Python and has used an LLM API before, but has not yet built a full RAG system end to end.

System Architecture, Component by Component

[Support Docs: HTML/PDF/Markdown]
        |
        v
[Ingestion Pipeline]
  - Document loader (per-format parser)
  - Chunker (structure-aware, see chunking decision below)
  - Embedder (batch API calls)
        |
        v
[Vector Store + Metadata Index]
  - Dense vectors (embedding model output)
  - BM25 sparse index (same chunk text, for hybrid search)
  - Metadata: source URL, doc title, last-updated timestamp
        |
        v
[Retrieval Layer]
  - Query embedding
  - Hybrid search: dense top-20 + sparse top-20, merged via
    reciprocal rank fusion
  - Re-ranker (cross-encoder) narrows merged set to top-5
        |
        v
[Generation Layer]
  - Prompt template: system instructions + retrieved chunks
    with source tags + user question
  - Citation enforcement: generator instructed to tag claims
    with [source: doc_id], validated post-hoc against
    retrieved chunk set
  - Fallback: "I don't have enough information" when
    retrieval confidence is below threshold
        |
        v
[Response + Citations returned to user]
        |
        v
[Feedback Capture: thumbs up/down + optional free text]
        |
        v
[Evaluation Harness: runs offline against labeled eval set
 before every deploy; also logs live feedback for drift
 monitoring]

Component Deep Dive: Ingestion Pipeline

The ingestion pipeline is where most portfolio projects lose credibility, because candidates default to a single fixed-size chunker regardless of document format. For a support corpus with a realistic mix of formats, build format-specific loaders:

• HTML help articles: parse with a structure-aware extractor (strip nav/footer, keep headers), chunk at H2/H3 boundaries.
• PDF manuals: extract text with layout awareness (a plain PDF-to-text extractor will interleave table cells incorrectly); chunk by detected section headers, and store table content as a separate chunk type with its own retrieval handling since tables retrieved out of context are close to useless.
• Chat transcript logs (if used as a knowledge source of resolved tickets): chunk per resolved conversation, not per message, since a single message out of context loses the resolution.

State the chunk size decision explicitly in the project write-up: 300-500 tokens per chunk for prose with 15% overlap is a defensible default, justified by the boundary-integrity and retrieval-precision reasoning covered in the companion chunking design review workbook on this site.

Component Deep Dive: Hybrid Retrieval

A project that uses vector search alone is the single most common gap that separates a mid-level portfolio project from a strong one. Dense retrieval alone fails on exact-match queries — a user asking about a specific error code or SKU number will often retrieve semantically-similar-but-wrong chunks, because dense embeddings are tuned for semantic similarity, not lexical exactness. Adding a sparse BM25 index over the same chunks and merging results with reciprocal rank fusion (RRF) recovers these exact-match cases without abandoning semantic search for paraphrased queries.

Concrete implementation: run both searches independently (dense top-20, sparse top-20), then score each chunk by RRF_score = sum(1 / (60 + rank)) across the two ranked lists it appears in, and take the top-5 by combined score. The constant 60 is a standard RRF smoothing value from the original RRF paper (Cormack et al.) and does not typically need tuning for a first implementation.

Add a cross-encoder re-ranker as the final step before generation. A cross-encoder scores each (query, chunk) pair jointly rather than comparing pre-computed independent embeddings, which produces meaningfully higher precision at the cost of latency — since it cannot be pre-computed and requires a forward pass per candidate pair at query time. This is why it runs on the narrowed top-20-ish candidate set from hybrid search rather than the full corpus: running a cross-encoder over the full corpus would not scale.

Component Deep Dive: Citation Enforcement

Generation without citation is the second gap that reads as unfinished in a portfolio review. Structure the prompt so each retrieved chunk carries a visible source tag ([doc_id: refund-policy-v3]), instruct the generator to cite the source tag next to each factual claim, and post-process the response to validate that every cited doc_id actually appears in the retrieved set — this catches hallucinated citations, which is a distinct failure mode from hallucinated facts and worth calling out separately in a project write-up, since a fabricated citation to a real-looking but non-existent document ID is a more dangerous failure than an uncited wrong answer, because it projects false confidence.

Add an explicit low-confidence fallback: if the top retrieved chunk’s hybrid score falls below a calibrated threshold (calibrate this against your eval set, not a guessed number), the generator returns “I don’t have enough information to answer this confidently” instead of forcing an answer from weak context. This single design decision is worth highlighting in interviews, since it demonstrates awareness that RAG systems fail silently by default and this is a deliberate mitigation.

Trade-offs Table: Architecture Decisions and What They Cost

Decision Rubric: What to Build First

If time is constrained (a weekend portfolio project rather than a multi-week build), build in this order, since each stage is independently demonstrable and interview-worthy even if you stop partway:

1. Format-aware ingestion + structure-based chunking (demonstrates document understanding, not just API calls).
2. Hybrid retrieval with RRF merge (demonstrates retrieval systems knowledge beyond “call the vector DB”).
3. Citation enforcement with validation (demonstrates production-safety thinking).
4. Cross-encoder re-ranking (demonstrates precision optimization).
5. Feedback loop and evaluation harness (demonstrates the operational mindset covered in the companion evaluation-plan page for this project).

Stopping after step 2 with a clear, honest write-up of what’s missing and why is stronger than a rushed step 5 that skips validation.

Deployment Considerations for a Portfolio Demo

A portfolio project needs to actually run somewhere a hiring manager can try it, not just exist as code in a repository. Keep the deployment footprint deliberately small: a serverless function or small container for the API layer, a managed vector database with a free tier (avoid running your own vector database cluster for a demo — that’s infrastructure overhead unrelated to the RAG skills you’re demonstrating), and a static frontend that calls the API. State the estimated monthly cost in the project README, since a reviewer skimming your GitHub will notice if a demo project appears to have an unbounded or unstated cost exposure — a small hard cap on daily API spend (implemented as a simple request counter) is worth adding and mentioning explicitly.

Rate-limit the public demo per IP or per session to avoid a viral post driving unexpected LLM API costs, and add a visible “this is a portfolio demo, not a production support system” disclaimer if you’re using real company documentation as the corpus, to avoid implying an affiliation that does not exist.

Observability: What to Log and Why

A project that logs nothing beyond the final response misses the chance to demonstrate operational maturity, which is a specific gap technical interviewers at infrastructure-focused teams probe for. At minimum, log per-request: the retrieved chunk IDs and their hybrid scores, the re-ranker’s output scores, whether the low-confidence fallback triggered, generation latency broken into retrieval time and generation time separately, and the user’s feedback signal (thumbs up/down) linked back to the original request. This log becomes the raw material for the evaluation harness’s drift monitoring and for debugging any specific bad response a reviewer or user flags — being able to say “here’s the exact retrieved context that produced this wrong answer” is a stronger interview story than “the system sometimes gets things wrong.”

Extending the Project: Signals That Impress Interviewers

Beyond the core five components, three extensions consistently read as senior-level judgment rather than checkbox feature addition: a query rewriting step that expands abbreviated or ambiguous user queries before retrieval (e.g., expanding “return window” using conversation history if the user previously mentioned a specific product); a multi-turn conversation handling layer that re-retrieves based on the full conversation context rather than just the latest message, since support conversations frequently build on prior turns; and a structured admission that the project has known limitations, with a specific list (for example: “the system does not currently handle documents added after the last ingestion run without a manual re-index trigger, tracked as a known gap”). This last one specifically counters a pattern interviewers flag negatively — projects presented as flawlessly complete, since real production RAG systems always carry known trade-offs, and a candidate who cannot name their own project’s limitations reads as less experienced, not more polished.

Relevant Book Sample

The 0→1 AI Engineer Interview Playbook (ASIN B0H2CML9XD) includes a chapter specifically on presenting portfolio projects in interviews — how to narrate this exact architecture in a way that surfaces the trade-off reasoning above rather than just listing the tech stack used. If you want deeper coverage of the reliability and feedback-loop side of this build (retry logic, drift monitoring, agent tool orchestration if you extend this project with agentic support workflows), the companion title The 0→1 Loop Engineering Playbook covers that ground; its ASIN is not yet publicly verified, so check the primary title’s listing page for current cross-references.

Read the project-presentation chapter: /go/B0H2CML9XD?source=ai-engineers-blog&page=aie-csrag-arch-001

Testing the Architecture Before Trusting It

Before treating any component above as done, run a targeted smoke test per stage rather than only testing the full pipeline end to end: feed the ingestion pipeline a document with a known table, and manually confirm the table survived chunking without being split mid-row; feed the retrieval layer a query with an exact SKU or error code, and confirm hybrid search returns it in the top results (a dense-only ablation run alongside it makes the value of the sparse index visible and quotable); feed the generation layer a query with no good matching document, and confirm the low-confidence fallback triggers instead of a confidently wrong answer. Each of these targeted tests catches a specific, named failure mode faster than waiting for it to show up buried inside an aggregate end-to-end evaluation score, and having them as a documented smoke-test checklist in the project repository is itself a piece of evidence worth mentioning in an interview.

Sources and Freshness

Architecture patterns here (hybrid search with reciprocal rank fusion, cross-encoder re-ranking, citation validation) reflect publicly documented RAG engineering practice as of 2026, drawing on the original RRF paper (Cormack, Clarke, Buettcher, SIGIR 2009) and standard cross-encoder re-ranking literature (e.g., Sentence-Transformers documentation). No proprietary company architecture is represented; this is a portfolio-project reference design. Last verified: 2026-07. Next review: quarterly.

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.