How It Works
AI models can only read a limited amount of text at once — the context window. A long report, a book chapter, or a two-hour meeting transcript will not fit, so you split the document into smaller pieces called chunks and feed them to the model one at a time (or store them in a vector database for retrieval-augmented generation). Chunking sounds trivial until you do it badly: cut in the middle of a sentence and the model loses the thread; make chunks too small and each one lacks the context to be useful; make them too big and you are back to the context-window problem. This tool splits your text three ways — by tokens, by characters, or by sentences — with a configurable overlap that keeps meaning intact across boundaries. All of it happens in your browser, so private documents never leave your device.
By tokens is the mode you want when the chunks are headed for a model. Token counts are estimated with the same calibrated heuristic used across this site's AI tools (about five percent average error against real tokenizers). The splitter walks through your text word by word, accumulating estimated tokens until the chunk reaches your size limit, and always breaks on a word boundary — never mid-word. By characters uses a simple sliding window: it takes size characters, then steps forward by size-minus-overlap, so consecutive chunks share exactly overlap characters. This mode is predictable and easy to reason about, and it is the right choice when your downstream system counts characters. By sentences splits on sentence boundaries and packs a fixed number of sentences per chunk — ideal when you want each chunk to be a complete thought, for example when summarizing paragraph by paragraph.
The overlap setting is what separates decent chunking from naive slicing. Without overlap, a sentence that straddles a chunk boundary gets cut in half, and neither chunk makes full sense. With a 50-token overlap, the end of each chunk repeats at the start of the next one, so boundary sentences survive whole. Overlap costs you a little duplication — the same text is embedded or processed twice — but for retrieval systems it measurably improves answer quality, because the chunk that contains the answer is more likely to contain the full sentence around it. Ten to twenty percent of the chunk size is the usual starting point: 50 tokens of overlap on 500-token chunks, for instance.
Context-window warnings
After splitting, the tool estimates the token count of every chunk and compares the largest one against the context windows of popular model tiers: 4K, 8K, 32K, and 128K tokens. If a chunk exceeds one of those limits, you get a warning naming which windows are too small. This is your signal to reduce the chunk size before sending the text to that model — a chunk the model cannot fit will be truncated, and truncation silently deletes the end of your content. Note that the warning is about individual chunks, not the whole document: the whole point of chunking is that the document can be arbitrarily long as long as each piece fits.
Downloads and formats
Each chunk can be copied individually with one click, or you can download everything at once. The TXT download writes the chunks as plain text separated by headers — handy for pasting into a chat in sequence. The JSON download gives you an array of objects with index, token estimate, character count, and text — the format most vector-database and embedding pipelines expect. The CSV download puts the same data in a spreadsheet-friendly table. Pick the format your next step consumes; all three contain exactly the same chunks.
Worked Example
Suppose you have a 12,000-word product manual — roughly 16,000 tokens — and you want to build a Q&A bot over it with a model that has an 8K context window. Your prompt template (system instructions plus the user's question) uses about 500 tokens, so each chunk must stay well under 7,500 tokens to leave room. A standard choice: 800 tokens per chunk with 100 tokens of overlap.
Paste the manual, select "By tokens," set chunk size to 800 and overlap to 100. The tool walks the text and produces about twenty-two chunks — each a whole number of words, each sharing roughly a paragraph with its neighbor. The overview cards confirm the totals: 22 chunks, about 16,000 tokens total, averaging around 730 tokens per chunk (slightly under 800 because chunks break on word boundaries). No context-window warning appears, because every chunk is far below the 8K limit. You download the JSON, feed each chunk's text to your embedding model, and store the vectors with the chunk index. When a user asks a question, you retrieve the three most similar chunks and stuff them into the prompt — about 2,400 tokens of context plus your 500-token template, comfortably inside the window.
Now consider a different job: summarizing a long interview transcript, section by section, in a chat interface. Here "By sentences" with 8 sentences per chunk and 2 sentences of overlap works beautifully — each chunk is a complete conversational beat, and the two-sentence overlap means the model sees how each beat connects to the previous one. You expand all, copy chunk one, ask for a summary, then continue down the list. Because the chunks never cut mid-sentence, the summaries stay coherent without you having to repair broken fragments by hand.
Choosing your numbers
There is no universal best chunk size, but there are well-tested starting points. For retrieval-augmented generation, 300 to 800 tokens per chunk with 10–20% overlap is the industry default — small enough that retrieved chunks are focused, large enough to contain complete ideas. For summarization, go bigger: 1,500 to 3,000 tokens, because summaries need broader context. For feeding a chat model sequentially, match the chunk to the task: one chunk per section, per scene, or per topic. When in doubt, start at 500 tokens with 50 overlap, test on your real data, and adjust once — chunking rewards one round of tuning far more than theorizing.
Tips and Common Mistakes
The classic mistake is chunking by characters with no overlap and a round number like 1,000. It feels precise, but it cuts sentences, words, and even paragraphs at arbitrary points, and retrieval quality suffers because half the chunks start mid-thought. If you must use character chunking — for example, because a downstream tool counts characters — always set an overlap of at least 100–200 characters, and prefer the token or sentence modes when the chunks are headed for a language model.
The second mistake is making chunks too small. Fifty-token chunks sound efficient, but a fifty-token fragment rarely contains a complete idea, so your retrieval system returns fragments that confuse the model instead of informing it. Extremely small chunks also multiply your embedding and storage costs, since every chunk is processed separately. Unless you have a specific reason (like fine-grained citation), stay above 200 tokens.
Third, people forget that overlap is not free. With 500-token chunks and 200 tokens of overlap, forty percent of your text is processed twice — doubling embedding costs and index size for a modest quality gain. The 10–20% guideline exists for a reason: enough to preserve boundary sentences, not so much that you are paying to store the document twice. Measure on your data if the workload is large; otherwise trust the default.
- Clean the text first. Strip navigation menus, page numbers, repeated headers and footers, and giant whitespace runs before chunking. Junk in means junk chunks — the splitter cannot tell boilerplate from content.
- Keep metadata with each chunk. The JSON download includes the chunk index; when you build your pipeline, also store the source document name and page or section number alongside it so answers can cite their sources.
- Watch the largest chunk, not the average. Averages hide outliers. The context-window warning checks the maximum chunk for exactly this reason — one oversized chunk can break an otherwise fine pipeline.
- Match chunking to the model's tokenizer. Token estimates here are within about five percent of real tokenizers, which is fine for sizing. For exact billing or hard limits, verify the largest chunk with the provider's own tokenizer or count-tokens endpoint.
- Test retrieval, not just splitting. Good-looking chunks can still retrieve poorly. Ask ten real questions, check whether the right chunks come back, and only then tune size and overlap — chunking is a means, not the goal.
One last practical note: this tool keeps your text entirely on your device, which makes it safe for confidential documents — contracts, medical notes, unreleased manuscripts. The chunks you download are yours to use anywhere, and nothing about the splitting process is sent to any server or AI provider.
Chunking for different document types
Not all documents chunk the same way. Code files do best with larger token chunks (800–1,200 tokens) because a function split across chunks is useless without its signature; consider chunking code by function or class boundaries rather than raw token counts when your pipeline allows it. Legal and technical documents with numbered sections work well with sentence or paragraph chunking that respects the section structure — never let a chunk cross a section boundary if you can avoid it, because the section heading is often the most informative sentence in the chunk. Conversational transcripts (meetings, interviews, support chats) benefit from sentence chunking with generous overlap, since speakers interrupt each other and thoughts span turns. Tables and spreadsheets are the hardest case: a row split across two chunks loses its column headers, so keep tables whole whenever possible, even if it means uneven chunk sizes. Whatever the document type, the workflow is the same — chunk, inspect the largest and smallest chunks, adjust, and re-chunk. Two minutes of inspection beats an hour of debugging a pipeline fed with broken pieces.