RAG Chunking: System Design

Answer-First Summary

A production chunking system is not a single function call — it is a pipeline stage with its own inputs, outputs, failure modes, and monitoring. Design it as four decoupled components: a document parser that preserves structure, a splitter that applies size and boundary rules per content type, a metadata enricher that attaches retrieval filters, and a re-indexing controller that handles updates without downtime. This article specifies each component’s interface and shows the architecture for a system handling mixed content types (prose, tables, code) at scale.

Scope Definition

Retrieval-Augmented Generation (RAG) is an architecture pattern where a system retrieves relevant text passages from a document store and injects them into a large language model’s (LLM) prompt to ground its output in source material. Chunking is the ingestion-time step that breaks source documents into retrievable units. This article treats chunking as a system component with defined interfaces, not as an interview talking point — it targets engineers who are building or operating the ingestion pipeline behind a RAG product, covering component boundaries, data flow, and operational concerns like re-indexing and monitoring. It assumes familiarity with vector embeddings (numerical representations of text used for similarity search) but explains every chunking-specific term inline.

Technical Framework: The Four-Component Pipeline

Component 1: Document Parser. Converts raw source formats (PDF, HTML, Markdown, DOCX, Confluence export) into a normalized intermediate representation that preserves structural signals: headings, list nesting, table boundaries, code block fences. This step is the single highest-leverage investment in chunking quality — a parser that flattens a PDF into raw text with no heading markers dooms every downstream chunking decision, because structure-aware splitting has nothing to split on. Output: a tree or flat list of typed blocks (heading, paragraph, table, code, list_item), each tagged with source page/line number for citation.

Component 2: Splitter. Consumes the typed block stream and applies per-type rules. Prose blocks get merged up to a token cap with sentence-boundary awareness. Tables are never split — they pass through as a single chunk regardless of size, because a partial table row is retrieval-useless. Code blocks split on function or class boundaries using a language-aware parser (an abstract syntax tree, or AST, walker) rather than token count, because a function cut mid-body breaks both readability and any downstream code-execution use case.

Component 3: Metadata Enricher. Attaches structured fields to every chunk before embedding: source document ID, section path (e.g., auth > oauth2 > refresh-tokens), content type, last-modified timestamp, and access-control tags if the corpus has permission boundaries. This metadata enables filtered retrieval — a query can be scoped to content_type: table or section_path starts_with: auth before the vector similarity search runs, which materially raises precision on narrow queries without shrinking chunk size.

Component 4: Re-indexing Controller. Manages the lifecycle of chunks as source documents change. This is the component most system-design answers omit and the one that separates a demo from a production system.

Worked Example: End-to-End Architecture for a Documentation RAG System

Scenario: a company wants to expose 5,000 internal Confluence pages (mixed prose, tables, embedded code snippets) through a RAG-based internal search assistant, updated as documents change throughout the day.

[Confluence API] --poll/webhook--> [Ingestion Queue]
        |
        v
[Document Parser]  -- typed block stream -->  [Splitter]
        |                                          |
        |                                          v
        |                                [Metadata Enricher]
        |                                          |
        |                                          v
   [Change Log DB]  <---------------------  [Embedding Service]
        |                                          |
        v                                          v
[Re-index Controller]                      [Vector Store]
        |                                          ^
        +------------------------------------------+
              (on doc change: delete stale chunk IDs,
               insert new chunk IDs, atomic swap)

Implementation detail for the re-indexing controller — the part most designs get wrong by re-embedding the entire corpus on any edit:

def reindex_document(doc_id: str, new_blocks: list, chunk_index: dict):
    """
    chunk_index maps doc_id -> list of chunk_ids currently live
    for that document, stored in a change-log table.
    """
    old_chunk_ids = chunk_index.get(doc_id, [])
    new_chunks = split_and_enrich(new_blocks, doc_id)
    new_chunk_ids = [c.id for c in new_chunks]

    # Only embed chunks whose content hash differs from the
    # previous version -- skip unchanged sections entirely.
    to_embed = [c for c in new_chunks if content_changed(c, old_chunk_ids)]
    embeddings = embed_batch(to_embed)

    vector_store.upsert(embeddings)
    stale_ids = set(old_chunk_ids) - set(new_chunk_ids)
    vector_store.delete(list(stale_ids))

    chunk_index[doc_id] = new_chunk_ids
    return {"embedded": len(to_embed), "deleted": len(stale_ids)}

The content-hash check is what makes this system operate at production scale: a single-paragraph edit to a 5,000-word document re-embeds one chunk, not the whole document, which keeps ingestion latency and embedding API cost proportional to the size of the edit rather than the size of the corpus.

Trade-Offs and Decision Matrix

Decision Rubric

1. If documents arrive with reliable structural markup (Markdown, HTML, Confluence storage format): invest in the parser component first — structural fidelity at parse time compounds through every downstream stage, while a weak parser cannot be compensated for by a smarter splitter.
2. If the corpus updates more than once per day: build the content-hash diffing re-indexing controller from day one. Retrofitting incremental re-indexing onto a full-reembed system is a larger engineering cost than building it correctly the first time.
3. If the product needs to answer “where did this come from” questions: the metadata enricher must carry source URL, page number, and section path — treat citation support as a system requirement, not a nice-to-have added later.
4. If mixed content types (tables, code, prose) coexist in the same corpus: route each type through type-specific splitting logic rather than a single universal splitter function; a universal splitter is the most common root cause of poor retrieval on structured content.
5. If query latency is the binding constraint: store full chunk text in the vector database payload to avoid a second read at query time, accepting the storage cost increase.
6. If storage cost is the binding constraint and query volume is moderate: store chunk IDs only and fetch text from the source-of-truth database at query time.

Scaling Considerations at Corpus Sizes Above 100K Documents

The four-component architecture above holds structurally at any scale, but three components need explicit scaling treatment once a corpus exceeds roughly 100,000 documents.

Parser throughput. A single-threaded parser processing PDFs or scanned documents can become the ingestion bottleneck well before the embedding service does, particularly for documents requiring OCR. Design the parser as a horizontally scalable worker pool consuming from the ingestion queue, with each worker stateless and idempotent — reprocessing the same document twice must produce the same typed block output, so that a worker crash mid-document does not corrupt the pipeline state.

Embedding service batching. Embedding APIs charge per token and typically offer meaningfully better throughput on batched requests than on one-chunk-at-a-time calls. Batch chunks from multiple documents into a single embedding request up to the provider’s batch size limit, but do not let batching delay latency-sensitive re-index operations — separate the ingestion queue into a bulk lane (large batches, higher latency tolerance) and a priority lane (small batches, low latency, used for user-triggered document edits that need to be searchable within seconds).

Vector store write amplification. At scale, the re-indexing controller’s delete-then-insert pattern for a single edited document can generate a disproportionate number of vector store write operations if chunk boundaries shift on every edit — a one-word change near the start of a long document can shift every subsequent chunk boundary in a naive fixed-token splitter, forcing a full re-embed of the rest of the document. Structure-aware chunking mitigates this because edits inside one section only shift chunk boundaries within that section, not the whole document; this is an underappreciated scaling argument for structure-aware chunking beyond its retrieval-quality benefits.

Monitoring the Chunking Pipeline in Production

A chunking system needs its own health metrics distinct from general application monitoring, because a chunking regression can degrade retrieval quality silently for weeks before anyone notices a drop in answer quality. Track at minimum:

• Chunk count per document over time. A sudden change in the average number of chunks produced per document after a parser or splitter deployment is a leading indicator of a regression — investigate before it reaches users.
• Chunk size distribution. Track the p50, p95, and p99 token count of chunks produced. A long tail of very small chunks (under 20 tokens) usually indicates a parser producing spurious empty or near-empty blocks; a long tail of oversized chunks indicates the splitter’s fallback logic is not triggering correctly.
• Re-indexing latency. Time from document-change event to the corresponding chunks being live and searchable in the vector store. This is a direct user-facing freshness metric for any product where users expect edits to be reflected quickly.
• Orphaned chunk rate. Chunks present in the vector store whose source document ID no longer exists in the source system — a signature of the re-indexing controller’s delete step silently failing. This metric should be checked on a recurring reconciliation job, not just at deploy time, since the failure mode it catches is a slow accumulation, not a sudden spike.

Testing the Chunking System Before Production Rollout

Treat the splitter and parser as testable units with the same rigor as any other data transformation code, independent of end-to-end retrieval evaluation. A minimal test suite for a chunking system includes: a fixed set of representative source documents (one per content type — prose, table-heavy, code-heavy, deeply nested headings) with hand-verified expected chunk boundaries; a regression test that re-runs this fixed set on every splitter or parser change and diffs the output against the last known-good chunking; and a load test that verifies re-indexing latency stays within its target bound as corpus size grows, since re-indexing performance degradation is often gradual and easy to miss until it becomes a user-visible freshness problem.

Security and Access Control in the Chunking Pipeline

Corpora with document-level or section-level access control (an internal knowledge base with confidential HR documents, for instance) require the chunking pipeline to propagate permission metadata correctly at every stage, not just at query time. A chunk that loses its source document’s access-control tags during parsing or splitting becomes a permission leak: it remains fully retrievable to any user whose query matches it semantically, regardless of whether that user should have access to the source document. The metadata enricher component must therefore treat access-control propagation as a required field, validated on every chunk before it is written to the vector store, with an ingestion-time failure (not a silent default to public) when a source document’s permission metadata cannot be determined. At query time, the retrieval layer must apply the access-control filter before or alongside the vector similarity search, never as an after-the-fact filter on results already shown to the requesting system, since a post-filter still means the sensitive chunk transited through more of the system than necessary.

Book CTA

The complete system design for a production RAG ingestion pipeline — including the vector index selection and reranking layers that sit downstream of chunking — is covered in The 0→1 AI Engineer Interview Playbook (/go/B0H2CML9XD?source=ai-engineers-blog&page=aie-rag-chunking-sysdes-002). Engineers coming from a data-pipeline or ML-infrastructure background who need the foundational data-engineering patterns this architecture builds on should also see The 0→1 Machine Learning Engineer Interview Playbook (/go/B0H256Z1MF?source=ai-engineers-blog&page=aie-rag-chunking-sysdes-002).

Sources and Freshness Note

This architecture reflects patterns documented in open-source RAG ingestion frameworks and vector database vendor reference architectures as of mid-2026. The content-hash diffing pattern for incremental re-indexing is a common production optimization, not a single vendor’s proprietary technique. Review this page quarterly, as vector database vendors continue to add native incremental-update APIs that may simplify the re-indexing controller design described here.

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.