RAG Reranking: Interview Answer Framework

Retrieval and reranking are frequently conflated by candidates who haven’t shipped a production RAG system. Interviewers ask reranking questions specifically to test whether you understand why a two-stage retrieve-then-rerank pipeline exists at all, rather than just retrieving more documents and passing them straight to the LLM.

Core Concepts

Retrieval (dense, sparse, or hybrid) is optimized for recall at speed — finding a reasonably good candidate set from millions of documents in milliseconds, using architectures (bi-encoders, BM25) that can pre-compute document representations independently of the query. Reranking is optimized for precision, using models that jointly process the query and each candidate document together, which is far more accurate but too slow to run against the full corpus.

The core reason cross-encoders aren’t used for first-stage retrieval: a bi-encoder computes document embeddings once, offline, and stores them for fast approximate nearest-neighbor lookup at query time. A cross-encoder must process the query and each document together, meaning there is no way to pre-compute anything — every candidate document must be freshly scored against the current query, at inference time, which is why cross-encoders only run against a small candidate set (typically top 20-100 from retrieval), never against the full corpus.

ColBERT and other late-interaction models sit architecturally between bi-encoders and cross-encoders: they compute token-level embeddings for both query and document independently (like a bi-encoder, so partially pre-computable), but score relevance via a MaxSim operation that compares every query token against every document token, capturing much of the cross-encoder’s precision at a fraction of the latency cost. This makes ColBERT-style models attractive for reranking larger candidate sets (hundreds rather than tens of documents) under tighter latency budgets.

Cohere Rerank is the most commonly cited managed reranking API in interviews — it exposes a simple API (query + list of documents in, relevance scores out) backed by a cross-encoder-quality model, removing the need to self-host a reranking model, at the cost of a per-query API fee and network latency.

📧 Get free interview prep resources — frameworks and real FAANG questions. Download the free kit →

Interview Answer Framework (4-Step)

Step 1 — Explain why a two-stage pipeline exists at all. State plainly: retrieval optimizes for speed across the full corpus (millions of documents, sub-second), reranking optimizes for precision across a small candidate set (tens to hundreds of documents, tens to hundreds of milliseconds). Trying to run a cross-encoder against the full corpus is computationally infeasible at any real scale — this is the foundational insight interviewers are listening for.

Step 2 — Size the candidate set. Propose retrieving top-50 to top-100 candidates from the retrieval stage (dense, sparse, or hybrid), then reranking down to the top-5 to top-10 that actually get passed into the LLM’s context window. State the reasoning: retrieval recall needs headroom (the right document should be somewhere in the top-50 even if not ranked first), and reranking’s job is to fix ordering errors within that set.

Step 3 — Choose a reranker based on the latency budget. If the system has a generous latency budget (batch or async use cases), a full cross-encoder or Cohere Rerank API call is fine. If the system is latency-sensitive (sub-500ms end-to-end target, live chat), a ColBERT-style late-interaction reranker or a smaller/distilled cross-encoder is the better tradeoff, since it recovers most of the precision gain at a fraction of the latency cost.

Step 4 — Tie it back to measured improvement. The strongest answers close with: “I’d measure NDCG@10 before and after reranking on a labeled eval set to confirm the reranking stage is actually improving ordering, since reranking adds latency and cost that should be justified by a measured lift, not assumed.”

Common Follow-ups

“How much does reranking typically improve retrieval quality?” In practice, reranking commonly lifts NDCG@10 or MRR by 10-30% relative to retrieval-only ordering, because the retrieval stage’s bi-encoder similarity is a coarser signal than a cross-encoder’s joint query-document attention. The exact lift is domain-dependent and should be measured, not assumed.

“What’s the latency cost of adding a reranking stage, and how do you budget for it?” A cross-encoder reranking 50 candidates typically adds 50-150ms to end-to-end latency depending on model size and batching. In a system with a 1-2 second end-to-end RAG latency budget (retrieval + reranking + LLM generation), this is usually acceptable; in a sub-300ms budget, a lighter or late-interaction reranker becomes necessary.

“Can you skip reranking if your retrieval is good enough?” Yes, and interviewers want you to say this rather than treating reranking as mandatory. If your retrieval-only NDCG is already high (e.g., because the corpus is small, homogeneous, or the embedding model is well-tuned to the domain), the marginal quality lift from reranking may not justify the added latency and cost — this is a call that should be validated empirically per system, not assumed as a default best practice.

Production Considerations

Reranking latency compounds with retrieval and generation latency in the overall RAG request path — a system design answer should always state the end-to-end latency budget and show how it’s allocated across retrieval, reranking, and generation, rather than discussing reranking in isolation.

Cost scales with candidate set size: reranking 100 documents costs roughly 2x reranking 50, since most rerankers process documents in the candidate set individually or in small batches. Tuning the candidate set size (how many documents come out of retrieval into reranking) is a real production lever for balancing quality against cost and latency, and should be tuned against measured NDCG, not set to an arbitrary round number.

Managed reranking APIs (Cohere Rerank, and similar offerings) remove the operational burden of hosting a cross-encoder model, but introduce a hard external dependency and per-query cost that scales with traffic — for high-volume production systems, self-hosting a smaller distilled cross-encoder or ColBERT-style model is often the more cost-effective long-term choice once volume justifies the infrastructure investment.

For a structured walkthrough covering embeddings, indexing, hybrid search, and reranking as one coherent RAG interview arc, see The 0-to-1 AI Engineer Interview Playbook (Amazon: https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20).

FAQ

Q: Do all RAG systems need a reranking stage? A: No. Reranking adds meaningful value when retrieval-only ordering has measurable precision errors within the top-k results, which is common with dense or hybrid retrieval at scale. For smaller, well-tuned corpora, measure NDCG with and without reranking before adding the complexity.

Q: How many candidates should retrieval pass to the reranker? A: A common production range is top-50 to top-100 from retrieval, reranked down to top-5 to top-10 for the LLM context window. The right number depends on your latency budget and measured recall — the true answer should come from your retrieval stage’s Recall@k curve, not a fixed convention.

Q: Is ColBERT a replacement for cross-encoder reranking? A: It’s a middle-ground alternative — ColBERT-style late-interaction models recover most of a cross-encoder’s precision at meaningfully lower latency, making them attractive when you need to rerank larger candidate sets under tight latency constraints. For the highest possible precision on small candidate sets with a generous latency budget, a full cross-encoder still edges it out.