Chunking: how documents get split

Imagine filing a book by cutting it into index cards. Cut through the middle of an explanation and the card becomes useless: half the reasoning stayed on the other one. Chunking is exactly that, and it decides what your system will be able to find.

It is the step people rush through in RAG, usually by leaving a library’s defaults untouched. It is also where most systems break: no embedding model, however good, retrieves a concept that was cut in half.

Why it matters more than the model

Document: a 50-page handbook

Bad chunking     chunks that cut through concepts
                 → retrieval finds half explanations
                 → the model answers halfway, confidently

Sane chunking    chunks that hold a whole concept
                 → retrieval finds complete explanations
                 → the model has something to work with

The important part: the failure is invisible. Nothing is reported, an answer comes back anyway, and the system just seems a bit dim. When a RAG setup “doesn’t work well” for no obvious reason, chunking is the first place to look.

The strategies, worst to best

Fixed size. Cut every N characters, ignoring the content.

"Polymorphism is a concept that lets | objects of different classes..."
                        cut here ↑

chunk 1: "Polymorphism is a concept that lets"
chunk 2: "objects of different classes..."      ← starts in mid-air

Simple and fast, but it cuts words in half and produces chunks that begin with no subject. It only makes sense on text with no structure at all, like log files.

With overlap. Same as above, but each chunk repeats a bit of the previous one’s tail.

|<------ chunk 1 ------>|
                 |<-- overlap -->|
                 |<------ chunk 2 ------>|

Typical overlap is 10-20% of the chunk size: on a thousand characters, one to two hundred. It limits the damage at the seams, does not solve long concepts, and in exchange grows the number of chunks you keep in the vector database.

By structure. Follows the document’s headings, sections and paragraphs: one chunk per section. Every chunk holds a complete thought, but sizes swing wildly — a three-line paragraph and an eight-page chapter become two very different chunks.

Recursive. The right starting point for almost everything. It tries the largest separator first and only goes finer while the chunk is still too long:

1. blank line   → separates paragraphs
2. newline      → separates lines
3. space        → separates words
4. character    → last resort

It respects the structure of the text when there is one, and still behaves predictably when there is not.

Format-aware. When you know what you are indexing, a splitter that understands the format beats everything: Markdown by heading, HTML by section, code by function or class, JSON by top-level object.

Strategy	Complexity	Quality	When
Fixed size	Minimal	Low	Text with no structure at all
With overlap	Low	Medium	Logs, chats, transcripts
By structure	Medium	High	Well-formatted documents
Recursive	Low	High	The sensible default
Format-aware	Medium	Very high	When the format is known

How big they should be

There is no right number, but there are two ways to get it wrong.

Too small (under two hundred tokens): “Polymorphism is a concept in object-oriented programming.” Retrieval finds it, but that chunk alone cannot answer anything: everything else is missing.

Too large (over two thousand): a whole chapter covering three topics. Its vector ends up being the average of all of them, so it is not particularly close to any question — and when retrieved it eats a large slice of the context window to say very little.

As orders of magnitude:

Size	Suits
~500 tokens	Definitions, FAQs, reference cards
~1000 tokens	Explanations, technical documentation, tutorials
~1500 tokens	Prose, long articles, manuals

One warning that costs people afternoons: characters are not tokens. If the library counts characters while you think in tokens, your chunks are shorter than you believe — by roughly a factor of four in English. Configuring the count in tokens is worth it, since tokens are the unit the model thinks in (the subject of the tokenization lesson).

Metadata is worth as much as the text

Every chunk should carry where it came from:

chunk:
  text: "Polymorphism lets objects of different classes..."
  source: java-handbook.pdf
  page: 145
  chapter: "5 - Object-oriented programming"
  section: "5.3 Polymorphism"

It serves four concrete purposes: citing the source in the answer, filtering the search to a single chapter, understanding which chunk produced a bad answer, and spotting duplicates when the same text arrives from two files.

There is a less obvious effect too: prefixing the chunk with the title of its section improves retrieval. A paragraph starting with “5.3 Polymorphism” sits closer, in embedding space, to a question about polymorphism than the same paragraph orphaned.

The same text, two different cuts

Take a page with three parts: a definition, a code example, a list of benefits.

Cut at a fixed 200 characters:

chunk 1: "## Polymorphism\nPolymorphism is one of the four pillars of"
chunk 2: " object-oriented programming. It lets objects of different classes"
chunk 3: " respond to the same message in different ways.\n\nThere are two kinds: overloadi"

The third chunk ends inside a word. None of these cards answers a question.

Cut recursively, with overlap:

chunk 1: definition + the two kinds of polymorphism
chunk 2: the complete code example
chunk 3: the benefits

Three cards, three whole concepts. Ask “show me an example of polymorphism” and retrieval can hand back exactly chunk 2.

Where it goes wrong

Leaving the defaults without looking at the chunks. The fix costs five minutes: print the first ten chunks and read them. If they make no sense on their own, they will make no sense to the model either.

Zero overlap. It is the starting value in several libraries, and it loses the sentences that straddle two chunks — precisely the ones that connect ideas.

One cut for every format. A handbook, a pile of conversations and a code archive do not split the same way. If you are indexing mixed material, pick the strategy per document type, not for the whole collection.

Re-indexing without emptying the store. You change the chunk size, run indexing again, and the old chunks stay: the store fills with two versions of the same text, and retrieval gets worse instead of better.

Tuning by feel instead of by question. The serious way to tune chunking is to take ten questions you already know the answers to, look at which chunks come back, and adjust. Without that, you are guessing.

In short

Concept	In one line
Chunking	Splitting documents into the pieces that will be retrieved
Why it matters	A concept cut in half cannot be retrieved, full stop
Sane default	Recursive, ~1000 tokens, 10-20% overlap
Too small	The chunk is found but cannot answer
Too large	The vector averages too many topics and is close to nothing
Metadata	Source, page, section: for citing, filtering and debugging
Useful trick	Prefix each chunk with its section title
How to tune it	With real questions, watching which chunks come back