RAG Query Rewriting: Interview Answer Framework

Retrieval-augmented generation systems live or die on the quality of the query that hits the vector store. Ask any engineer who has shipped RAG to production and they will tell you: the raw user question is almost never the best possible search query. This is why query rewriting has become one of the most frequently tested topics in AI engineering interviews — it separates candidates who have read about RAG from candidates who have actually debugged a retrieval pipeline at 2am.

This article gives you a repeatable framework for answering query rewriting questions, whether the interviewer asks a conceptual question (“What is HyDE?”) or a system design question (“How would you improve retrieval recall for a legal search product?”).

Core Concepts

Query rewriting is the umbrella term for any technique that transforms a user’s raw input before it is embedded and used to search a vector index or hybrid retrieval system. The core insight is simple: users write short, ambiguous, conversational queries, but the documents you’re retrieving are written in a completely different register — formal, dense, jargon-heavy. Closing that gap is the job of query rewriting.

The unifying theme across all four techniques: you are using an LLM as a preprocessing step to close the vocabulary and reasoning gap between what a user types and what a retriever needs to find the right chunk. None of these techniques replace good chunking or good embeddings — they compensate for the fact that even great embeddings can’t fix a badly phrased query.

📧 Get free interview prep resources — frameworks and real FAANG questions. Download the free kit →

Interview Answer Framework

When an interviewer asks about query rewriting, use this four-step structure to keep your answer tight and demonstrate both depth and production judgment.

Step 1 — Name the failure mode first. Don’t jump straight to HyDE. Start by stating the problem you’re solving: “The core issue is that user queries and corpus documents live in different linguistic registers, so cosine similarity between the raw query embedding and document embeddings often misses relevant chunks even when the content is a perfect match.” This shows you understand why the technique exists, not just its name.

Step 2 — Pick the right technique for the failure mode. Map the specific symptom to the specific fix:

• If the query is too short or vague → HyDE (generate a hypothetical answer, embed that).
• If the query requires background knowledge or multi-hop reasoning → step-back prompting (abstract to a more general question first).
• If the query is actually multiple questions in one → multi-query decomposition (split, retrieve per sub-query, merge/dedupe).
• If the query uses different vocabulary than the corpus → LLM query expansion (add synonyms, expand acronyms).

Step 3 — Discuss the tradeoff explicitly. Every rewriting technique adds latency (one or more extra LLM calls before retrieval even starts) and cost. State this directly: “HyDE roughly doubles latency to first retrieval because you need an LLM round-trip before you can even call the vector store. In a chat product with a 2-second SLA, that’s a real constraint.” Interviewers want to see you weigh recall gains against latency and cost — not blindly recommend the fanciest technique.

Step 4 — Close with a production caveat. Mention that these techniques should be evaluated with a retrieval eval set (recall@k, MRR) before and after rewriting, because rewriting can sometimes hurt precision by drifting from user intent. A strong closing line: “I’d A/B this against a baseline with a golden query set before shipping it, because HyDE in particular can hallucinate details that pull in irrelevant documents.”

Common Follow-ups

Interviewers will almost always probe deeper after the initial answer. Be ready for:

• “How do you combine multi-query decomposition with reranking?” — Explain that after retrieving for each sub-query, you typically get overlapping and sometimes contradictory chunks, so you dedupe by document ID, then pass the merged candidate set through a cross-encoder reranker before final context assembly.
• “What happens if HyDE hallucinates a wrong fact in the hypothetical document?” — Acknowledge this is a real risk. The hypothetical document is only used for its embedding, never shown to the user or fed into the final generation step, which limits the blast radius. But if the hallucinated content pulls in the wrong topic area entirely, retrieval quality suffers.
• “How do you decide when NOT to rewrite the query?” — Good answer: short, well-formed, keyword-heavy queries (like a product SKU or exact error code) usually don’t benefit from rewriting and can even be hurt by it, since rewriting adds paraphrase noise. Use a lightweight classifier or heuristic (query length, presence of quotes/exact terms) to route around rewriting for these cases.
• “How would you evaluate whether query rewriting is actually helping?” — Recall@k and MRR against a labeled eval set, plus online metrics like answer groundedness and user reformulation rate (if users keep re-asking, your retrieval is failing).

Production Considerations

In production, query rewriting is rarely a single monolithic step — it’s a small pipeline stage with its own latency budget, caching layer, and fallback path. A few things that separate a toy implementation from a production one:

• Latency budgeting. Every rewriting technique adds an LLM call before the retrieval call even starts. Teams running latency-sensitive chat products often use a small, fast model (not the main generation model) specifically for query rewriting to keep this step under 200-300ms.
• Caching rewritten queries. If your product sees repeated or near-duplicate queries (common in support/FAQ use cases), cache the rewritten form keyed on a normalized version of the input to skip the LLM call entirely on cache hits.
• Fallback and timeout handling. If the rewriting LLM call times out or errors, fall back to raw-query retrieval rather than failing the whole request. Rewriting should be a quality enhancement, not a single point of failure.
• Combining with hybrid retrieval. Query rewriting is most powerful when combined with hybrid search (dense + BM25/sparse), since expansion techniques like LLM query expansion often improve keyword-based recall as much as semantic recall.
• Guardrails against drift. Log rewritten queries alongside originals and periodically audit for cases where the rewrite drifted from user intent — this is the single most common silent failure mode in production RAG systems.

If you want a deeper structured walkthrough of how to talk through RAG system design end-to-end — not just query rewriting but chunking, indexing, reranking, and evaluation — check out The 0-to-1 AI Engineer Interview Playbook (Amazon: https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20), which walks through exactly this kind of interview framework in more depth across the whole RAG stack.

FAQ

Q: Is HyDE always better than a raw query embedding? A: No. HyDE tends to help most on sparse, ambiguous, or open-ended queries where the vocabulary gap between question and answer is large — think “what should I do about a slow database” versus a document titled “PostgreSQL index tuning guide.” For precise, keyword-heavy queries (exact error messages, product names, SKUs), HyDE can actually hurt by introducing paraphrase drift. Always validate with an eval set rather than assuming it universally helps.

Q: How is step-back prompting different from multi-query decomposition? A: Step-back prompting generates one more abstract version of the question and retrieves using both the original and the abstracted version — it’s about zooming out for context. Multi-query decomposition splits one question into multiple parallel, independent sub-questions and retrieves for each separately — it’s about breaking a compound question into pieces. They solve different problems and can be combined.

Q: Does query rewriting replace the need for good chunking and embedding model selection? A: No, and this is a common interview trap. Query rewriting compensates for the mismatch between query and document phrasing, but it can’t fix bad chunk boundaries (chunks that split a fact across two chunks) or a poorly suited embedding model. A strong interview answer treats query rewriting as one layer in a stack that also includes chunking strategy, embedding model choice, hybrid retrieval, and reranking — not a silver bullet on its own.