RAG Cost Optimization: Interview Answer Framework

Every AI engineering interview in 2026 eventually asks some version of “your RAG system works, but it’s too expensive at scale — what do you do?” This question tests whether you understand that RAG cost is a multi-line-item problem, not a single lever. This article gives you a framework to answer it precisely.

Why This Question Is Now Standard

In 2024 and 2025, most RAG interview questions focused on getting retrieval to work at all. By 2026, with LLM API costs still a meaningful line item for any product doing millions of queries a month, interviewers assume you can build a working RAG pipeline and instead probe whether you can make one economically sustainable. The candidates who stand out name concrete levers with concrete numbers, not generic advice like “use a cheaper model.”

The Answer Framework: CUTS

Structure your answer around four cost levers, in the order you’d actually attack them:

• C — Cache aggressively. Embedding and response caching eliminate redundant compute for repeated or near-duplicate queries.
• U — Utilize tiered retrieval. Not every query needs your most expensive retrieval path; route by query complexity.
• T — Trim token budgets. Control context window size deliberately instead of stuffing in every retrieved chunk.
• S — Screen for duplicates. Deduplicate queries and documents upstream so you never pay to process the same thing twice.

Embedding Caching: The First Lever

Embedding generation is a fixed cost you pay once per unique text, so the single highest-leverage optimization is making sure you never re-embed the same content twice. This applies in two directions:

• Document-side caching: hash each chunk’s content and store the embedding keyed by that hash. When documents are re-ingested (common in pipelines that re-scan a corpus nightly), skip re-embedding unchanged chunks entirely.
• Query-side caching: cache embeddings for frequently repeated or templated queries (e.g., “summarize this ticket” patterns in a support tool), with a TTL appropriate to how often the underlying corpus changes.

A concrete number to cite in an interview: if 30% of incoming queries are near-duplicates of previously seen queries (common in customer support and internal knowledge-base tools), query embedding caching alone can cut embedding API spend by a proportional amount with near-zero quality risk.

Query Deduplication vs. Tiered Retrieval

Interviewers like this comparison because it shows you understand these levers attack different cost lines (embedding vs. generation vs. compute overhead) rather than treating “cost optimization” as one undifferentiated bucket.

Tiered Retrieval in Practice

Tiered retrieval means routing queries to a retrieval-and-generation path scaled to their actual complexity, instead of always running your most expensive pipeline. A practical three-tier design:

1. Tier 1 — Cache/FAQ hit. If a semantically similar query has a cached, human-verified answer, serve it directly with no LLM call.
2. Tier 2 — Lightweight retrieval + small model. Simple factual queries get a fast vector search (small k) and a cheaper, smaller LLM for generation.
3. Tier 3 — Full pipeline. Complex, multi-hop, or ambiguous queries get full hybrid retrieval, reranking, and your flagship model.

A router — often a small classifier or even a cheap LLM call — decides tier assignment. The key interview insight: the router itself must be cheap, or you’ve just added another cost line without savings. A distilled classifier or a rules-based heuristic (query length, presence of comparison words, prior escalation history) is usually enough; you don’t need a frontier model to route.

Token Budget Management

Many teams over-retrieve by default — pulling top-10 or top-20 chunks “to be safe” — and pay for tokens that don’t improve answer quality. A disciplined approach:

• Set an explicit token budget per query type (e.g., 2,000 tokens of context for simple FAQ, 6,000 for multi-document synthesis).
• Rerank before truncating, so you drop the least relevant chunks first, not the ones that happened to be retrieved last.
• Measure the marginal quality gain of each additional chunk on your eval set — teams are often surprised that chunk 6 through 10 add cost without moving faithfulness or accuracy scores.
• Compress context where possible: summarizing lower-priority chunks instead of including them verbatim can preserve signal at a fraction of the token cost.

A Sample Interview Answer, End to End

“I’d start by measuring where the cost actually goes — embedding, retrieval infra, or generation tokens — because the fix is different for each. If a meaningful share of queries are repeats or near-duplicates, I’d add semantic query caching first since it’s the highest ROI, lowest risk change. Then I’d introduce tiered retrieval: a router sends simple queries to a lightweight path with a smaller model and shallow retrieval, and reserves the full hybrid-retrieval-plus-flagship-model pipeline for genuinely complex queries. Finally, I’d audit token budgets per query type and rerank-then-truncate instead of stuffing in every retrieved chunk, since marginal chunks past the top few often don’t move faithfulness scores but do move the bill.”

Common Mistakes Candidates Make

• Jumping straight to “use a smaller model” without addressing retrieval-side waste first — model swaps alone rarely solve cost problems holistically.
• Not distinguishing between embedding cost, retrieval infra cost, and generation cost as separate line items with separate fixes.
• Proposing caching without addressing staleness — a cache with no invalidation strategy becomes a correctness bug, not just a cost fix.
• Ignoring the cost of the router itself in tiered systems.
• Failing to mention measurement: cost optimization without before/after metrics is not a defensible engineering practice.

Practice Prompts

• “Your RAG system costs 3x more per query than budgeted. Walk me through your diagnosis process.”
• “How would you design a tiered retrieval system without hurting answer quality for complex queries?”
• “What’s your strategy for embedding cache invalidation when the source documents change silently?”

Further Reading

This article is part of a series on RAG interview frameworks covering observability, latency, hallucination detection, and production deployment. For a complete, structured resource covering the full range of AI engineering interview topics, see The 0-to-1 AI Engineer Interview Playbook (Amazon: https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20).

Key Takeaways

RAG cost optimization interview answers should attack four distinct levers — caching, tiered retrieval, token budgets, and deduplication — in an order that reflects risk-adjusted ROI, not just theoretical maximum savings. Candidates who name specific mechanisms, specific trade-offs, and specific failure modes (stale caches, under-served complex queries, router overhead) consistently outperform candidates who offer only generic cost-cutting platitudes.