RAG chunking: Interview Answer Framework

Answer first

When an interviewer asks “how would you chunk documents for a RAG system,” they are testing whether you treat chunking as a search-quality lever, not a preprocessing afterthought. The strong answer: pick chunk size and boundary strategy based on the retrieval unit that maximizes answer-bearing precision for the query distribution, not a fixed token count. Start from 200-500 tokens with semantic or structure-aware boundaries (headings, sentences, code blocks), overlap 10-15% to preserve cross-boundary context, and validate with a retrieval eval before shipping any single number. There is no universal “right” chunk size; there is only a chunk size validated against your corpus and your query set.

Scope and assumptions

This page covers text-heavy RAG systems (docs, support tickets, contracts, code comments) retrieved via dense or hybrid search and fed into an LLM context window. It assumes:

• You control ingestion (you can re-chunk and re-index on demand).
• The corpus is not fully structured data (that’s a different problem: SQL/API retrieval).
• You have or can build a small labeled eval set (20-100 query/relevant-passage pairs) — without this, “best chunk size” claims are unfalsifiable and interviewers should be skeptical of a candidate who skips this.

Out of scope: chunking for fine-tuning data prep, and multi-modal (image/table) chunking, which need separate frameworks.

Core framework: the chunking decision tree

Walk the decision in this order — this is what a strong candidate narrates out loud:

1. What is the retrieval unit that answers a typical query?

• Single fact / definition → small chunks (100-300 tokens), higher precision, less noise in context.
• Multi-step procedure or reasoning chain → larger chunks (400-800 tokens) or parent-child retrieval (retrieve small, expand to parent section).
• Code → chunk by function/class boundary, never by fixed token count (a chunk split mid-function is close to useless).
• Tables → keep tables atomic; never split a table across chunks.

2. What is the boundary strategy?

3. What overlap prevents context loss at boundaries?

10-15% overlap is a reasonable starting default. The purpose is narrow: prevent a sentence that spans a boundary from losing its antecedent. Overlap is not a substitute for correct boundary placement — over-relying on overlap to paper over bad splitting inflates index size and retrieval noise without fixing the root cause.

4. Do you need parent-child (small-to-big) retrieval?

Retrieve small chunks for precision (embedding match quality is higher on focused text) but pass the parent section/document to the LLM for context sufficiency. This decouples the retrieval unit from the generation unit and is usually the highest-leverage single change teams make after they discover chunking is hurting recall.

Worked example

Scenario: A 40-page internal API reference doc, being indexed for a RAG-based dev-support assistant. Query: “What status code does the /orders endpoint return on a duplicate idempotency key?”

Naive approach (400-token fixed windows, no structure awareness): the endpoint description, the status code table, and the idempotency section land in three different chunks with no reliable overlap, because the doc uses nested subheadings the splitter ignores. Retrieval returns the endpoint description chunk (highest embedding similarity to “orders endpoint”) but misses the status code table two subsections down. The LLM either hallucinates a status code or says “not found.”

Corrected approach:

1. Parse the doc’s Markdown/HTML structure; chunk by ##/### section boundary, capping at ~500 tokens, splitting only sections that exceed the cap.
2. Keep each status-code table as one atomic chunk with its parent heading prepended as a text prefix (“Section: /orders — Idempotency handling”) so the embedding captures the heading context even though the table content is generic-looking.
3. Set retrieval to top-8 chunks at 300-token granularity, then expand each hit to its full parent section (up to ~1,200 tokens) before passing to the LLM, capped at 3 expanded sections to control context budget.
4. Result on the 60-query eval set: recall@8 (fraction of queries where the answer-bearing chunk is retrieved) rose from 61% to 89%; the idempotency query above now retrieves the correct table because the heading-prefixed chunk boundary keeps it a first-class retrieval unit.

This is the shape of answer interviewers want: a specific failure mode, a specific mechanism that fixes it, and a specific measured before/after — not “I’d use recursive chunking with overlap.”

Trade-offs and failure modes

• Chunks too small: high precision per chunk, but you lose surrounding context and pay more retrieval calls/tokens to reassemble meaning. Symptom: correct chunk retrieved, but LLM answer is incomplete or contradicts itself because it lacks the qualifying sentence one chunk over.
• Chunks too large: embedding similarity gets diluted (a 1,500-token chunk about five different sub-topics matches everything a little and nothing well). Symptom: precision@k drops, and the LLM has to find the needle inside a chunk that’s mostly irrelevant, wasting context budget.
• Semantic chunking on noisy/OCR’d text: embedding-similarity breakpoints assume coherent local topic structure; on noisy text (scanned PDFs, transcripts with disfluencies) it over-fragments and adds ingestion latency without a retrieval quality payoff. Fall back to structural or fixed-size chunking and measure before investing in semantic splitting.
• Chunking code by fixed token count: near-guaranteed failure mode — always chunk by AST/function boundary for code, and say so explicitly if the interview scenario involves a code-search RAG system.
• Ignoring the query distribution: a chunk size tuned for “define X” queries will underperform on “compare X and Y” queries which need multiple entities co-located. If the product supports both query types, either chunk at two granularities and merge results, or bias toward the larger unit and accept some precision loss.

Decision rubric

Use this as the answer’s closing structure — most interviewers are explicitly listening for a rubric, not just narrative:

1. Default to structure-aware chunking (headers/AST) over fixed-size whenever the source has native structure. Only fall back to fixed-size recursive splitting for unstructured prose.
2. Start chunk size at 300-500 tokens, overlap at 10-15%, and treat both as tunable parameters, not constants — state this explicitly to signal you know they need validation.
3. Always build a retrieval eval set (even 20-30 hand-labeled query/passage pairs) before claiming a chunk size is “correct.” An answer with no evaluation step is a yellow flag in a senior interview.
4. Decouple retrieval granularity from generation granularity via parent-child/small-to-big retrieval when queries need more context than a precision-optimized chunk can hold.
5. Treat tables, code blocks, and other atomic structures as non-splittable units regardless of the size policy.
6. Re-chunk is cheap; re-embedding a full corpus is not free — mention cost/latency of re-indexing as a constraint when proposing frequent strategy changes in production.

Follow-ups interviewers ask next

• “How would you evaluate whether a chunk size change actually helped?” → recall@k and precision@k on the labeled eval set, plus an end-to-end answer-correctness eval (LLM-as-judge or human) since retrieval metrics alone don’t guarantee better final answers.
• “What if the corpus is multi-lingual?” → tokenizer behavior differs by language; token-count-based chunk sizes need per-language calibration, and sentence-boundary detection needs language-aware sentence splitters, not naive regex on periods.
• “How do you handle chunking for a corpus that updates hourly?” → incremental re-chunking scoped to changed documents/sections only, with a versioned index or blue-green index swap to avoid serving stale-and-fresh chunks from the same document simultaneously.

Framework and examples above are hypothetical illustrations for interview preparation, not reports of an actual candidate interview or employer feedback.

Handling the Curveball: When the Interviewer Pushes Back

Strong interviewers do not let a clean framework stand unchallenged — they will follow up with a pointed objection to see whether your reasoning holds under pressure. Three pushback patterns come up repeatedly, and preparing a specific response to each is worth more than memorizing the base framework alone.

Pushback 1: “Why not just use the largest chunk size the embedding model’s context window allows?” The temptation is to think bigger chunks are strictly safer because they lose less context. The correct response names the precision cost directly: an embedding vector represents a fixed-dimensional summary of everything in the chunk, and a chunk covering five unrelated subtopics produces a vector that is a blurred average of all five, which degrades similarity matching against any single-topic query. Cite the concrete mechanism — vector dilution — rather than restating “it depends,” which sounds evasive to an interviewer who is testing whether you understand why size matters, not just that it does.

Pushback 2: “Your eval set is only 30 queries — how do you know that generalizes?” This is testing statistical reasoning, not chunking knowledge. The right response acknowledges the limitation honestly (30 labeled queries gives a noisy recall estimate, especially for rare query types) and states the mitigation: stratify the eval set across the known query type distribution (narrow lookup vs. broad synthesis), expand the eval set incrementally as production query logs accumulate, and treat early recall numbers as directional rather than final.

Pushback 3: “What if the document has no structure at all — just raw scraped web text?” This tests whether your framework is dogmatic (always prefer structure-aware chunking) or adaptive. The correct answer falls back to sentence-window chunking with a slightly higher overlap ratio (20-25% instead of 10-15%) to compensate for the loss of structural signal, and flags that unstructured scraped text is also a candidate for semantic chunking if the added ingestion cost is justified by the stakes of the retrieval task.

A Second Worked Example: Contract Review RAG

Interviewers sometimes probe whether your framework generalizes beyond documentation by switching domains mid-conversation. A useful second example to have ready: a RAG system over a corpus of signed vendor contracts, where queries are things like “what is the termination notice period in the Acme Corp contract” or “which contracts have an auto-renewal clause.”

This domain changes two things about the chunking decision. First, clause-level structure matters more than section-level structure — a single contract section can contain several independently queryable clauses (termination, liability, renewal), so chunking at the section level would bundle unrelated clauses into one embedding, the same dilution problem described above but at a finer grain. The fix is to chunk at the clause or numbered-paragraph level, using the contract’s own numbering scheme as the structural boundary rather than markdown headers, since legal documents carry structure in numbered clauses rather than heading tags.

Second, the cost of a missed retrieval is asymmetric and high — a missed termination clause in an automated contract-review tool has real legal and financial consequences, which changes the calculus on overlap and redundancy. In this domain, err toward higher overlap (20-25%) and toward returning more chunks per query (top-8 to top-10 instead of top-3 to top-5) even at the cost of some retrieval noise, because the cost of the LLM synthesizing an answer from an incomplete retrieval set is higher than the cost of it filtering a slightly noisy one. Naming this asymmetric-cost reasoning explicitly is what turns a generic chunking answer into one that demonstrates judgment about when to deviate from default parameters.

Relevant book sample

The 0→1 AI Engineer Interview Playbook (ASIN: B0H2CML9XD) includes a full RAG system design chapter that walks this same chunking decision tree alongside vector index selection, hybrid search, and evaluation design — the four sub-questions interviewers chain together in a single “design a RAG system” prompt. The chapter’s worked example extends this exact API-docs scenario into the retrieval, reranking, and generation stages that typically follow the chunking question in a real system-design round.

If your target role leans more general ML infrastructure than applied LLM engineering, The 0→1 Machine Learning Engineer Interview Playbook (ASIN: B0H256Z1MF) covers the adjacent data-pipeline and feature-store design questions that RAG ingestion pipelines share with classical ML systems.

See the RAG chunking chapter and worked example on Amazon →

Sources and freshness

• Chunking strategy comparisons synthesized from publicly documented framework behavior (LangChain RecursiveCharacterTextSplitter, LlamaIndex node parsers) as of 2026-07.
• Recall@k figures in the worked example are illustrative, computed against a hypothetical 60-query eval set for demonstration of methodology, not a specific production system’s measured results.
• Page reviewed: 2026-07-15. Next freshness review: 2026-10-15 (quarterly, per site content-writing spec).

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.