· ai-engineers Editorial · Career · 5 min read
Vector Search Vs Keyword Search Hybrid Retrieval
BM25 vs embeddings vs hybrid retrieval: benchmarks, cost tradeoffs, and interview-ready system design answers.
Why Retrieval Architecture Is Now a Core Interview Topic
By mid-2026, every AI engineering interview loop touching RAG, search, or agent tooling includes a retrieval design question. The reason is simple: production RAG systems that rely purely on vector similarity are hitting a wall on exact-match queries — part numbers, error codes, legal citations, SKUs — while pure keyword search misses semantic paraphrase. Interviewers use this topic because it exposes whether a candidate understands tradeoffs rather than just calling an API.
If you’re prepping for onsite loops, this is one of the highest-leverage topics to master, alongside behavioral and coding rounds. The 0-to-1 AI Engineer Interview Playbook (https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20) devotes an entire chapter to retrieval system design because it recurs across FAANG, unicorns, and Series B startups alike.
How Keyword Search (BM25) Actually Works
BM25 (Best Matching 25) is a probabilistic ranking function built on term frequency-inverse document frequency (TF-IDF) principles. It scores documents based on:
- Term frequency saturation: repeated terms add diminishing returns, controlled by parameter k1 (typically 1.2–2.0)
- Document length normalization: parameter b (typically 0.75) penalizes long documents that naturally contain more term matches
- Inverse document frequency: rare terms across the corpus carry more weight than common ones
BM25 is implemented in Elasticsearch, OpenSearch, and Lucene-based systems. It requires no training, no embedding model, and no GPU — just an inverted index. Query latency at p99 is typically under 20ms for corpora up to tens of millions of documents.
Where BM25 wins: exact identifiers, acronyms, proper nouns, code snippets, numeric ranges, and any query where the user already knows the exact term they’re looking for.
Where BM25 fails: synonym gaps (“car” vs “automobile”), cross-lingual queries, and conceptual queries where the user describes an idea rather than naming it.
How Vector (Dense) Search Works
Dense retrieval encodes both queries and documents into fixed-dimension embeddings (commonly 384–3072 dimensions) using models like OpenAI’s text-embedding-3-large, Cohere embed-v4, or open-weight options like BGE-M3 and Nomic Embed. Similarity is computed via cosine distance or dot product, typically served through approximate nearest neighbor (ANN) indexes such as HNSW, IVF, or ScaNN.
Key production considerations as of July 2026:
- Embedding dimensionality tradeoffs: Matryoshka Representation Learning (MRL) now lets teams truncate embeddings from 3072 to 256 dimensions with under 3% recall loss, cutting storage and query cost dramatically.
- Index build time: HNSW indexes over 10M+ vectors can take hours to build; incremental insertion adds latency variance that interviewers ask about directly.
- Recall@k degrades under domain shift: embeddings trained on general web text underperform on legal, medical, or internal jargon-heavy corpora without fine-tuning or adapter layers.
Where vector search wins: paraphrase matching, cross-lingual retrieval, conceptual/semantic queries, and multi-modal retrieval (text-to-image, image-to-text).
Where vector search fails: exact string matches, rare tokens not well-represented in training data, and queries requiring precise numeric or boolean filtering.
Hybrid Retrieval: The 2026 Production Standard
Almost no serious production RAG system in 2026 relies on a single retrieval method. Hybrid retrieval combines BM25 and dense vectors, then fuses rankings using one of these approaches:
- Reciprocal Rank Fusion (RRF): combines ranked lists from both retrievers using
1/(k + rank), with k typically set to 60. No score normalization needed, which makes it robust and simple to implement. - Weighted linear combination: normalizes BM25 and cosine scores, then combines with a tunable alpha (commonly alpha=0.5 as a starting point, tuned via offline eval).
- Cross-encoder reranking: retrieve top-50 to top-100 candidates from both retrievers, then rerank with a cross-encoder (e.g., Cohere Rerank 4, bge-reranker-v3) that jointly scores query-document pairs. This adds 20-80ms latency but consistently lifts nDCG@10 by 8-15% in published benchmarks.
Comparison Table
| Dimension | Keyword (BM25) | Vector (Dense) | Hybrid + Rerank |
|---|---|---|---|
| Exact match accuracy | Excellent | Poor-Fair | Excellent |
| Semantic/paraphrase recall | Poor | Excellent | Excellent |
| Infra cost | Low | Medium-High | High |
| p99 latency | 5-20ms | 15-60ms | 60-150ms |
| Cold-start (no training data) | Works immediately | Needs embedding model | Needs both |
| Best for | Logs, code, IDs | Conceptual search, chat | Production RAG |
| Common index | Inverted index | HNSW/IVF | Both + reranker |
Interview Framing: What Evaluators Actually Listen For
When an interviewer asks “design a retrieval system for our support docs,” they are scoring you on these signals:
- Do you ask about query distribution first? Exact-ID lookups vs conceptual questions changes the whole architecture.
- Do you mention evaluation metrics? Recall@k, nDCG@10, MRR — not just “it works well.”
- Do you discuss cost at scale? Vector search cost scales with corpus size and query volume; candidates who ignore this get dinged in senior-level loops.
- Do you know reranking exists? Many mid-level candidates stop at “combine BM25 and vectors” without mentioning the reranking stage that actually drives the accuracy lift.
- Do you discuss chunking strategy? Retrieval quality is bottlenecked by chunk size and overlap long before it’s bottlenecked by the retrieval algorithm.
FAQ
Q: Should I always use hybrid retrieval, or is vector-only ever sufficient? A: Vector-only is sufficient for pure conversational or conceptual search where users never type exact IDs or codes — think general knowledge chatbots. The moment your corpus includes product SKUs, error codes, or proper nouns your embedding model wasn’t fine-tuned on, hybrid becomes necessary. Most production systems in regulated or technical domains (support, legal, healthcare) default to hybrid by 2026.
Q: What’s the single most common mistake candidates make in retrieval system design interviews? A: Jumping straight to “use a vector database” without asking clarifying questions about query patterns, latency budget, and corpus size. Senior interviewers actively want to see you push back and scope the problem before proposing an architecture.
Q: How do I practice this topic before an onsite? A: Build a small hybrid retrieval pipeline (Elasticsearch or OpenSearch + a vector DB like Qdrant or pgvector) over a public dataset, measure recall@10 with and without a reranker, and be ready to talk through the numbers. The 0-to-1 AI Engineer Interview Playbook (https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20) includes a worked system-design script for exactly this scenario, with the follow-up questions interviewers tend to ask.
Retrieval architecture questions aren’t going away — if anything, as more companies ship RAG and agentic systems in 2026, this topic is becoming as standard as “reverse a linked list” was a decade ago. Know the tradeoffs cold, have real numbers to cite, and you’ll separate yourself from candidates who only know the buzzwords.