AI

By the end of this build he has implemented, from scratch in Python, a working byte-pair-encoding tokenizer capable of reproducing GPT-4's own tokenization behavior — training a vocabulary, encoding text to tokens, and decoding back — and used that implementation to explain a long list of otherwise-mysterious LLM failures, from bad arithmetic to a glitch token that makes models swear at users. The throughline is that tokenization is a completely separate preprocessing stage from the language model itself, with its own training set and algorithm, and that a surprising fraction of what looks like a model or architecture problem actually traces back to this stage.

Why not just feed in raw bytes or characters. His earlier character-level tokenizer (from the "build GPT from scratch" lecture) used a vocabulary of only 65 symbols, one per character seen in a small dataset — simple, but impractical at scale. Text in Python is a sequence of Unicode code points, of which roughly 150,000 exist, but using code points directly is unattractive both because the standard keeps changing and because encoding to raw UTF-8 bytes gives a vocabulary of only 256 symbols, which would stretch every document into extremely long byte sequences that a Transformer's finite context window and attention cost can't accommodate efficiently.

The byte-pair encoding algorithm. The fix is to compress those byte sequences by repeatedly finding the most frequently occurring adjacent pair of tokens and minting a new token to represent it, replacing every occurrence — for example, merging the common pair "e" + " " (space) into a single new token once it's identified as the single most frequent pair in a sample paragraph. Repeating this some chosen number of times (a hyperparameter — GPT-4 does enough merges to reach a vocabulary near 100,277) builds up a "forest" of merges on top of the 256 base byte tokens, and this merge table, once built, is all that's needed to both encode new text into tokens and decode tokens back into text. He demonstrates this concretely: 20 merges on a ~24,000-byte blog excerpt compress it to about 19,000 tokens, a 1.27x ratio that improves further with more merges. Critically, vocabulary size can't grow without limit — a bigger vocabulary means a bigger embedding table and a bigger final classification layer, more parameters that individual rare tokens will under-train, and a risk of cramming too much meaning into single tokens for the network's fixed per-token computation to unpack.

GPT-2's regex-based pre-splitting. OpenAI didn't apply plain BPE to raw text; GPT-2's released code (encoder.py) first splits text using an elaborate regex that separates letters from numbers from punctuation from whitespace, so that merges can never cross those boundaries — preventing nonsensical joint tokens like "dog!" merging as a unit with "dog?" while still keeping the two semantically identical. He steps through the regex clause by clause, showing quirks like inconsistent handling of apostrophes (a Unicode right-single-quote isn't matched the way a plain ASCII apostrophe is) and case-sensitivity bugs GPT-2's own documentation admits to. GPT-4's tokenizer (used via the tiktoken library, vocabulary roughly double GPT-2's ~50k) revised this pattern to be case-insensitive and, notably, to group runs of whitespace into single tokens rather than leaving each space isolated — a change he shows cuts token counts for indented Python code roughly in half, and identifies as a real contributor to GPT-4's markedly better code handling, separate from any model or training improvement.

Special tokens and model surgery. Beyond byte-derived tokens, tokenizers reserve dedicated IDs for structural markers — GPT-2's single <|endoftext|> token delimits documents during pretraining, while chat models add a family of tokens (<|im_start|>, <|im_end|>, and others) to mark turns between user and assistant. Adding a new special token to a pretrained model requires literal "model surgery": extending the embedding table with a new randomly-initialized row and widening the final output layer by one, which is routine practice when moving a base model toward a fine-tuned chat or tool-using model.

SentencePiece as an alternative. Used by Llama and Mistral, SentencePiece differs fundamentally from tiktoken by running BPE merges directly on Unicode code points rather than on UTF-8 bytes; any code point too rare to have a token gets mapped to a fallback — either a generic unknown token, or, if "byte fallback" is enabled (as in Llama 2), decomposed into UTF-8 bytes and represented by dedicated byte tokens instead. He walks through SentencePiece's large, historically-accumulated configuration surface (sentence-splitting assumptions, normalization options, a mandatory unknown token, a "dummy prefix" space added to make a word look the same whether or not it starts a sentence) and judges it considerably less clean than tiktoken's approach, though it remains widely used because — unlike tiktoken — it supports training a vocabulary, not just running inference with one.

Multimodal tokenization and vocabulary extension. He notes briefly that the same Transformer architecture is increasingly used across modalities simply by inventing new tokenizers — chunking images into patches (citing an early image-tokenization paper and OpenAI's Sora, which represents video as "visual patches" analogous to text tokens) rather than by changing the model itself. Separately, techniques like "gist tokens" show vocabularies can be extended purely to compress long prompts: new tokens are trained by distillation, with the rest of the model frozen, so a handful of learned tokens can later stand in for an entire long prompt with near-identical model behavior.

Diagnosing real failures through tokenization. He closes by tracing a list of well-known LLM quirks back to this stage. Long, high-frequency tokens like .DefaultCellStyle bundle many characters into one indivisible unit, so a model asked how many "L"s it contains gets the answer wrong — it never sees individual letters, only the whole chunk — while asking it to first print the characters separated by spaces and then reverse them works, because that breaks the token apart. Arithmetic suffers similarly: digit groupings into tokens are arbitrary and inconsistent (a four-digit number might be one, two, three, or four tokens depending on what merged), disrupting the character-level carrying logic addition needs — which is why Llama 2 explicitly splits all digits during tokenization. Non-English text tokenizes far less efficiently since the tokenizer's own training data skews English — a Korean greeting can take three tokens versus one for its English equivalent, a roughly 3x expansion that effectively shrinks the usable context window for other languages. Oddities like a completion API breaking on trailing whitespace, or choking on the literal string "<|endoftext|>", stem from the mismatch between how partial or special tokens are represented internally versus how they appear in raw input. The most vivid example is "SolidGoldMagikarp": a Reddit username that appeared often enough in the tokenizer's own training corpus to earn a dedicated token, but never occurred in the language model's actual training data — leaving that embedding row permanently untrained and effectively random, so invoking the token later triggers undefined behavior (evasion, hallucination, even insults), which he compares to reading uninitialized memory in a C program. He ends with a practical recommendation: reuse GPT-4's existing tokenizer and tiktoken library where possible, and be wary of SentencePiece's many easily-miscalibrated settings if training a custom vocabulary is unavoidable.