Working in a Google Colab notebook, he builds a decoder-only Transformer from an empty file up to a working character-level Shakespeare generator — the same architecture underlying ChatGPT, minus its fine-tuning stages — training a roughly 10-million-parameter model on the ~1-million-character "tiny Shakespeare" dataset until validation loss falls from an untrained ~4.87 down to 1.48, at which point its output starts to look like recognizable (if nonsensical) Shakespearean-formatted text. The build closely mirrors his own nanoGPT repository, and he closes by mapping every piece back onto GPT-3's actual scale and onto what ChatGPT adds on top.
Data and the simplest possible model. The dataset is tokenized at the character level — a deliberately simple choice, contrasted with GPT's byte-pair or SentencePiece subword tokenizers, which trade a larger vocabulary for shorter sequences. Text becomes a single long tensor of integers, split 90/10 into train and validation sets, and sampled in fixed-length chunks (block size) so the model sees every context length from one token up to the maximum, which matters later during generation. The first working model is a bigram: each token, in isolation, uses an embedding table lookup to directly predict logits for the next token, with no communication between positions. Trained with Adam, this baseline's loss falls from about 4.7 to roughly 2.5 — proof the optimization loop works, but the output is gibberish because each character predicts the next based only on its own identity.
The mathematical trick behind attention. To let earlier tokens influence a later prediction without expensive looping, he shows that multiplying a token sequence by a row-normalized lower-triangular matrix computes a running causal average in one matrix multiplication — averaging in a token's own vector with all preceding ones while never looking ahead. Expressed as a masked softmax (zeroing future positions to negative infinity before normalizing), this same operation becomes the scaffold for self-attention once the uniform triangular weights are replaced by learned, data-dependent ones.
Self-attention itself. Each token emits three vectors via separate linear projections: a query ("what am I looking for"), a key ("what do I contain"), and a value (what actually gets aggregated once another token finds it interesting). Attention weights come from the dot product of every query against every key, masked to block future positions, then softmaxed and used to weight-sum the values — meaning tokens dynamically decide, based on their own content, how much to attend to each earlier token, rather than always averaging uniformly. A critical detail: the raw dot products are divided by the square root of the head size before the softmax, because otherwise their variance grows too large at initialization, driving softmax toward near one-hot outputs that let each token learn from effectively only a single other token. He's careful to frame attention as a general communication mechanism over a directed graph, not something intrinsically tied to language — the causal, future-blind mask is specifically what makes it a decoder rather than the fully-connected encoder used in tasks like sentiment classification, and cross-attention (queries from one source, keys/values from another) is what lets an encoder-decoder system like the original machine-translation Transformer condition generation on a separately-encoded input sequence.
Multi-head attention and computation. Running several smaller attention heads in parallel and concatenating their outputs — analogous to grouped convolutions — lets different heads specialize in different relationships (e.g., a vowel token searching specifically for nearby consonants) rather than forcing one attention pattern to do everything; this dropped validation loss from 2.4 to 2.28. Because attention only lets tokens communicate, without giving them time to "think" about what they gathered, a small per-token feedforward MLP (linear → ReLU → linear) is added after each attention step, further improving loss to 2.24.
Why depth needs residual connections and LayerNorm. Simply stacking multiple communicate-then-compute blocks made the network start to degrade rather than improve, a classic deep-network optimization problem. The fix, borrowed directly from the 2015 ResNet paper, is residual (skip) connections: each block forks off the main pathway, transforms, and adds its output back in, so gradients have an unimpeded "highway" back to the input during backpropagation, with each block's contribution initialized near zero and growing in over training. Paired with LayerNorm (which, unlike BatchNorm, normalizes each token's own feature vector rather than across the batch) applied before rather than after each sub-layer — a deliberate deviation from the original paper's post-norm design, adopted because it works better in practice — this let him stack six full blocks and reach a validation loss of 2.06, with dropout added to control overfitting as the model grew.
Scaling up and comparison to GPT-3. With block size raised to 256, embedding dimension to 384, six attention heads, six layers, and dropout at 0.2 — about 15 minutes of training on an A100 GPU — validation loss reaches 1.48, and generated text starts to visually resemble the structure of the source play. He then sets this ~10-million-parameter model, trained on roughly 300,000 GPT-vocabulary-equivalent tokens, against GPT-3's actual specs: architecturally near-identical hyperparameter choices, but scaled to 175 billion parameters trained on 300 billion tokens — a roughly millionfold increase, requiring thousands of coordinated GPUs rather than one.
Why pretraining alone isn't ChatGPT. A model trained only this way is a "document completer": given a question, it may just continue with more questions or drift into unrelated text, because it has only learned to imitate whatever an internet document containing that text tends to look like. Turning that into an assistant requires a separate, mostly non-public alignment pipeline he did not implement here: supervised fine-tuning on a smaller set of curated question-and-answer-formatted conversations, followed by training a reward model from human rankings of candidate responses, followed by using that reward model to further tune the policy via PPO reinforcement learning — the process OpenAI describes in its ChatGPT blog post, and which nanoGPT itself only covers up through the pretraining stage.