RAG Chunking: Failure Modes and Debugging

The Symptom You See First

A Retrieval-Augmented Generation (RAG) system that retrieves documents from a vector store and feeds them into a large language model (LLM) prompt is failing in production: users report answers that are wrong, incomplete, or contradictory, but the model itself has not changed and the embeddings have not changed. In the majority of these cases, the root cause traces back to chunking — the step that splits source documents into retrievable units before embedding. This article works backward from six observable failure symptoms to the chunking defect that causes each one, with a debugging procedure for each.

Scope Definition

This piece is a diagnostic reference, not an introduction to chunking concepts. It assumes you already have a working RAG pipeline and are troubleshooting degraded retrieval or generation quality. Each section names a symptom an engineer or support team would actually report, traces it to a specific chunking defect, and gives a concrete debugging step to confirm the diagnosis before applying a fix. It does not cover embedding model selection or LLM prompt engineering defects — those produce different symptom signatures covered elsewhere.

Failure Mode 1: “The Answer Is Missing a Detail That’s Definitely in the Docs”

Symptom: The LLM’s response is directionally correct but omits a specific number, date, or condition that a human reading the source document would immediately find.

Root cause: The fact lives in a chunk that was not among the top-k results returned by the retriever, because the chunk boundary separated the fact from the surrounding context that made it embed-relevant to the query. This happens most often when fixed-token-count chunking splits mid-paragraph, landing a qualifying clause (“except when the account is on the enterprise tier”) in a different chunk than the rule it modifies.

Debugging procedure:

1. Locate the source chunk containing the missing fact by searching the raw indexed chunks directly (not through the RAG pipeline — inspect the vector store payload).
2. Check whether that chunk was returned in the top-k retrieval results for the failing query. If it was retrieved but the LLM still omitted the fact, the defect is in generation, not chunking — stop here and redirect debugging.
3. If it was not retrieved, inspect the chunk’s boundaries: does it start or end mid-sentence, or is the qualifying clause separated from the rule it modifies by a chunk break?
4. Confirm by manually embedding the isolated fact-bearing chunk and comparing its cosine similarity to the query against the similarity of the chunk that was actually retrieved. A meaningfully lower similarity score confirms the boundary placement is the defect.

Fix: Increase overlap between adjacent chunks (10-20% of chunk size) so qualifying clauses near a boundary appear in both chunks, or switch from fixed-token to sentence-aware or structure-aware splitting so boundaries fall at natural breaks rather than arbitrary token counts.

Failure Mode 2: “Retrieval Returns Chunks That Are Technically Relevant but Useless”

Symptom: The top-k retrieved chunks all mention the right keywords, but none of them contains a complete, self-sufficient answer — the LLM has to guess or hedge.

Root cause: Chunks are too small relative to the complexity of the question being asked. A 150-token chunk optimized for narrow fact lookup cannot hold a multi-step procedure or a comparison between two entities, so even perfect retrieval returns fragments instead of answers.

Debugging procedure:

1. Classify the failing queries: are they asking for a single fact, or a synthesis across multiple facts (comparisons, multi-step procedures, “explain how X and Y interact”)?
2. If synthesis-style queries dominate the failures, check average chunk token count against query complexity — chunks under 300 tokens rarely hold enough context for synthesis questions.
3. Confirm by manually concatenating the top-5 retrieved chunks and checking whether the concatenation (not any single chunk) contains a complete answer. If yes, the defect is chunk size, not retrieval ranking.

Fix: Implement parent-child retrieval — retrieve small chunks for precision, but expand each retrieved hit to its full parent section before passing it to the LLM. This decouples the granularity used for search from the granularity used for generation.

Failure Mode 3: “Table Data Comes Back Garbled or Incomplete”

Symptom: Queries about tabular data (pricing tiers, comparison tables, status code references) produce answers that mix up rows or invent values.

Root cause: The document parser flattened a table into plain text without preserving row/column structure, and the splitter then cut the flattened table text at an arbitrary token boundary, separating a row’s label from its value.

Debugging procedure:

1. Pull the raw chunk(s) containing the table from the vector store and read them as plain text.
2. Check whether table structure survived: are rows and columns still visually distinguishable, or is it a run-on string of numbers and labels?
3. Check whether the table was split across multiple chunks — search for the table’s header row and its data rows separately; if they are in different chunk IDs, this confirms the defect.

Fix: Treat every table as an atomic, non-splittable chunk regardless of its token length, and serialize it with explicit structure preserved (markdown table syntax or a labeled key-value format) rather than flattening rows into a single run of text.

Failure Mode 4: “Retrieval Quality Degraded After a Content Update”

Symptom: A RAG system that worked well for weeks starts returning stale or duplicate information after a batch of source document edits.

Root cause: The re-indexing process either failed to remove stale chunks tied to the old version of an edited document, or embedded the new document without deleting the superseded chunk IDs, leaving both old and new versions live in the vector store simultaneously.

Debugging procedure:

1. Query the vector store directly for all chunks tagged with the affected document ID.
2. Check for duplicate or conflicting chunks referencing the same source section with different content — this is the signature of an incomplete re-index.
3. Check the ingestion pipeline logs for the affected document’s last update: did the delete-stale-chunks step execute, or only the insert-new-chunks step?

Fix: Make chunk deletion and chunk insertion atomic for a given document during re-indexing — either both succeed or neither does — and maintain a chunk-ID index per source document so stale IDs can be reliably identified and removed.

Failure Mode 5: “Code Search Returns Broken Snippets”

Symptom: A code-search RAG feature returns function bodies that are cut off mid-function, missing the closing brace or the return statement.

Root cause: Code was chunked using the same fixed-token-count splitter used for prose, which has no awareness of function or class boundaries.

Debugging procedure:

1. Pull the raw chunk and check whether it starts or ends at a function/class definition boundary or mid-body.
2. Confirm the splitter configuration used for this content type — if it is the same generic text splitter used for documentation prose, that is the defect.

Fix: Route code content through a language-aware splitter that parses the abstract syntax tree (AST) and chunks at function or class boundaries, never at a fixed character or token count.

Failure Mode 6: “Retrieval Precision Is High but Answer Accuracy Is Still Low”

Symptom: Manual review confirms the correct chunk is consistently in the top-3 retrieval results, yet the LLM still produces wrong answers.

Root cause: This is the one failure mode on this list where chunking is likely NOT the defect. When retrieval precision is confirmed high, the defect has moved downstream — into prompt construction, context ordering, or the LLM’s handling of conflicting information across multiple retrieved chunks.

Debugging procedure:

1. Confirm retrieval precision empirically (don’t assume): log the top-k chunk IDs for 20-30 failing queries and manually verify the correct chunk is present.
2. If retrieval precision is confirmed above roughly 80%, stop investigating chunking and redirect debugging effort to the prompt template and generation stage.

Decision Rubric: Which Failure Mode Matches Your Symptom

Failure Mode 7: “The Same Query Returns Different Answers on Different Days”

Symptom: Users report that asking an identical question produces materially different answers across separate sessions, with no source document changes to explain the shift.

Root cause: This is most often a re-indexing race condition where the vector store briefly contains both the old and new chunk versions of a recently edited document during the delete-then-insert window, and which version gets retrieved depends on timing rather than content correctness. A less common but real cause: non-deterministic chunk ID generation on re-index (e.g., IDs derived from insertion order rather than content hash), which causes the same logical chunk to receive a different ID on each re-index pass, breaking any downstream caching or deduplication logic keyed on chunk ID.

Debugging procedure:

1. Confirm the answer variance correlates with a recent re-index event on the relevant document, not with genuinely different underlying content.
2. Check whether chunk IDs for the affected document are stable across successive re-index runs of unchanged content — regenerate the chunk index twice in a row on a test document and diff the resulting chunk IDs.
3. If IDs are unstable, this confirms non-deterministic ID generation as the root cause; if IDs are stable but answers still vary, investigate the re-indexing controller for a non-atomic delete-then-insert sequence.

Fix: Generate chunk IDs deterministically from a hash of the source document ID, section path, and content — not from insertion order or a random UUID — so that unchanged content always produces the same chunk ID across re-index runs, and make the delete-and-insert step atomic (or use an upsert operation where the vector database supports it) to eliminate the race window.

A Systematic Debugging Checklist Before Escalating to “It’s a Model Problem”

Teams under pressure to ship often misattribute chunking-caused failures to “the LLM being unreliable” and respond by trying a larger or different model, which rarely fixes a retrieval-layer defect. Before concluding the model is the problem, work through this checklist:

1. Isolate retrieval from generation. Manually run the failing query against the vector store directly, bypassing the LLM entirely, and inspect the raw top-k chunks returned. If the correct information is not present in those chunks, no model change will fix the symptom — this is unambiguously a chunking or retrieval defect, not a generation defect.
2. Check chunk boundaries around every fact the answer needs. For a query requiring multiple facts, verify each fact-bearing sentence is intact within a single chunk or reliably spans an overlap region — a partial fact split across a hard boundary with no overlap is invisible to both retrieval and generation.
3. Check for duplicate or conflicting chunks. Search the vector store for multiple chunks referencing the same source section, which indicates a re-indexing defect (see Failure Mode 4) rather than a chunking-boundary defect.
4. Compare retrieval scores between the correct chunk and the chunks actually returned. If the correct chunk’s similarity score is close to the returned chunks’ scores, this points to a genuine retrieval ranking issue that additional reranking infrastructure may address. If the correct chunk’s score is far lower, the more likely defect is that the chunk’s content or boundary placement makes it a poor semantic match for the query as written — a chunking, not ranking, problem.
5. Only after ruling out the above, investigate generation. Check whether the LLM had access to the correct chunk in its context and still produced a wrong answer — this is the only scenario where the defect genuinely lives downstream of chunking and retrieval.

This checklist matters because chunking defects and model defects produce superficially similar symptoms (wrong or incomplete answers), but the fixes are entirely different, and teams that skip straight to model-level fixes (bigger model, different prompt, more few-shot examples) on a chunking-caused failure burn significant engineering time without resolving the underlying issue.

Book CTA

A structured walkthrough of RAG failure diagnosis — including the retrieval and generation-stage failure modes adjacent to the ones covered here — appears in The 0→1 AI Engineer Interview Playbook (/go/B0H2CML9XD?source=ai-engineers-blog&page=aie-rag-chunking-fmd-003), which uses debugging scenarios like these as system-design interview material. For the underlying data-pipeline debugging discipline this diagnostic approach borrows from, see The 0→1 Machine Learning Engineer Interview Playbook (/go/B0H256Z1MF?source=ai-engineers-blog&page=aie-rag-chunking-fmd-003).

Sources and Freshness Note

These failure modes are synthesized from commonly reported RAG production issues documented across open-source RAG framework issue trackers and vector database vendor troubleshooting guides as of mid-2026. Debugging procedures describe a general diagnostic method, not a specific vendor’s tooling. Review this page quarterly as chunking tooling and RAG framework defaults continue to evolve.

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.