{caption}
}
Try it yourself at tiktokenizer.vercel.app.
## How a model "reads" When you read English fluently, you don't sound out every letter. You see "the" and "have" and "tokenization" as single shapes, and your brain pulls up the meaning in one go. The rare and unfamiliar parts, like "antidisestablishmentarianism" or "solidgoldmagickarp", you slow down and parse in chunks: *anti-dis-establish-ment-arian-ism* or *solid-gold-magic-karp*. A tokenizer does roughly the same thing for an LLM. Common strings get a single ID, rare ones get broken into a handful of subword pieces, and anything weirder than that falls back to individual bytes. Every model's vocabulary is fixed and determined by the tokenizer—once the tokenizer is trained, nothing more about it is learned at inference time. The trade-off between token length and vocabulary size is straightforward: Using characters as tokens give you a tiny vocabulary but absurdly long sequences, and the model has to re-derive "h-e-l-l-o means hello" every single time. Using whole words as tokens gives you short sequences but a vocabulary of millions, including separate tokens for typos, novel words, URLs, and rare names. Subwords strike a sweet spot between these two extremes: models typically have \~50,000 to 200,000 tokens in their vocabulary, so every possible input is representable, but common text stays short. A rough rule of thumb for English is that one token ≈ 4 characters ≈ ¾ of a word. So 1,000 tokens is roughly 750 words, or one short page. ## What tokenization actually does Tokenization is a two-step process: first, splitting the text into chunks, then looking each chunk up in a table to get an integer ID. ```text theme={null} "Hello, world!" → ["Hello", ",", " world", "!"] → [9906, 11, 1917, 0] ``` Notice three things about this example: * `Hello` and ` world` (with a leading space) are each one token. The space matters and travels with the word. * `,` and `!` are their own tokens. Punctuation almost always is, because it's common enough to warrant its own token ID. * The IDs are lookup indices into a fixed vocabulary. There's no math here, no learning. The same text always produces the same IDs, every time. Different models use different tokenizers, so the same string can produce different IDs across model families like GPT-5.5, Claude Opus 4.8, and DeepSeek-V4. Within one model family, the tokenizer is typically fixed and baked in at pretraining time. ## Byte pair encoding (BPE) Almost every modern tokenizer is built using an algorithm called **byte pair encoding (BPE)**. The training procedure is as follows: 1. Start with a vocabulary of every distinct byte (or character). 2. Across the training corpus, count every pair of adjacent tokens. 3. Take the most common pair, merge it into a new token, and add it to the vocabulary. 4. Re-tokenize the corpus using the new vocabulary. 5. Repeat until the vocabulary reaches the target size (typically anywhere between 50,000 and 200,000 unique tokens). The list of merges is saved in order. At inference time, encoding a new piece of text means applying the same merges in the same order. Fast, deterministic, no model required.