Andrej Karpathy

This is a practical tour, not a build: a walk through how to actually use commercial LLM products as of early 2025, illustrated entirely with the presenter's own daily queries across ChatGPT, Claude, Gemini, Grok, Perplexity, and NotebookLM. By the end the viewer has a mental map of text chat, reasoning models, tool use (search, code execution, deep research), file and multimodal input/output, and the quality-of-life features (memory, custom instructions, custom GPTs) that separate casual use from power use — plus a habit of treating every model output as a probabilistic first draft to verify, not a fact.

The model as a zip file. Everything starts from one mental image: talking to ChatGPT means streaming tokens into and out of a fully self-contained neural network, with no calculator, browser, or tools by default. Pretraining is described as compressing the entire internet into a lossy, probabilistic "zip file" — for a trillion-parameter model, roughly a terabyte — by training it to predict the next token in web documents; this is expensive (tens of millions of dollars, months of compute) and infrequent, which is why models carry a knowledge cutoff. Post-training then swaps in human-written conversation data to give that same network the style of a helpful assistant — attaching, in his words, a smiley face to the zip file. A worked example: asked about caffeine in an Americano, the model's answer (63mg) is trustworthy mainly because the fact is common and unlikely to have changed, not because the model is authoritative. Practical advice follows directly: start a new chat when switching topics, since irrelevant tokens sitting in the context window both distract the model and slightly slow and raise the cost of generation; and always check which model and pricing tier you're on, since free tiers often mean a smaller, dumber model (GPT-4o mini vs. GPT-4o, Claude Haiku vs. Sonnet) without saying so.

Thinking models. A third training stage, reinforcement learning, lets a model practice on large sets of math and code problems and discover its own effective thinking strategies — backtracking, revisiting assumptions — resembling human inner monologue rather than anything a labeler could hand-write. He illustrates the payoff with a real bug: his gradient check was failing, and non-thinking GPT-4o's advice missed the actual issue. Switching to o1 pro ($200/month tier), the model thought for about a minute and correctly identified a parameter-packing mismatch. He replicated the same prompt across providers: Claude 3.5 Sonnet solved it without any dedicated thinking mode, Grok 3 solved it, and DeepSeek R1 (run through Perplexity, which surfaces its raw chain-of-thought) worked through the bug over several minutes before also finding it. The takeaway is to reach for a thinking model specifically for hard math/code/reasoning problems, since it adds little for something like travel advice, and to check whether a given provider even ships one (Anthropic did not, at the time of filming).

Internet search. Since the zip file's knowledge stops at its training cutoff, fresh questions require the model to emit a special token that triggers a live web search, pulling page text into the context window before answering. His running example is checking the release date of a new White Lotus season: without search, no model could know; with it, ChatGPT, Grok, and Gemini 2.0 Flash answered correctly with citations, while Claude 3.5 and Gemini 2.0 Pro lacked the integration entirely at the time. He treats Perplexity as his default search tool for anything time-sensitive or niche — market holidays, filming locations, product rumors, "what's this tweet storm about" — always checking citations since the underlying answer can still be hallucinated.

Deep research. Combining search and extended thinking, this ChatGPT Pro feature (then $200/month, with Perplexity and Grok offering rough equivalents) spends up to tens of minutes issuing many searches and reading many sources before writing a long, cited report. Asked to research a longevity-supplement ingredient, ChatGPT returned a report citing 27 sources after five minutes of work; a request for a table of U.S. LLM labs and their funding, by contrast, came back with real errors — xAI missing entirely, Hugging Face and EleutherAI wrongly included — underscoring that even a heavily-sourced report is a first draft requiring independent checking, not a finished answer.

Working with documents. Beyond search, models can take concrete documents directly into their context window via upload. He demonstrates dropping a 30-page biology paper (on a DNA foundation model called Evo 2) into Claude and ChatGPT to get summaries and ask questions, and describes routinely reading books this way — pasting chapters of Adam Smith's 1776 Wealth of Nations into Claude alongside the reading, asking for a chapter summary first and then following up with questions, which he says dramatically increases retention, especially for unfamiliar fields or old texts. No tool yet makes this seamless, he notes; it's still copy-paste back and forth.

Code execution and Claude artifacts. Giving a model a Python interpreter lets it write and actually run programs rather than guess arithmetic in its head — demonstrated by a large multiplication where Grok 3 hallucinated a wrong answer while ChatGPT and Claude wrote code and got it right. In "Advanced Data Analysis," ChatGPT is shown researching and plotting OpenAI's valuation history, fitting a trend, and extrapolating to 2030 — where it silently assumed a missing 2015 value and then reported $1.7 trillion for 2030 when its own code, once reprinted, actually computed roughly $20 trillion, a discrepancy only caught by rereading the code. Claude's separate "artifacts" feature is shown generating a working flashcard React app from pasted text and, more usefully in his own workflow, turning a book chapter into a Mermaid-diagram concept map — both cases of the model not just answering but building and running a small interactive tool inline.

Vibe coding. For real software work he leaves the browser entirely for Cursor, an editor built around Claude 3.7 Sonnet that has full context of a project's files. A short live demo has Cursor's "composer" agent build a Tic-Tac-Toe app from a five-sentence request, then add a confetti effect and a victory sound entirely on its own initiative, pausing only to confirm running a shell command — what he terms "vibe coding," where you mostly supervise rather than write code, with the fallback of reading and editing the files yourself when needed.

Audio, images, and video. He distinguishes "fake audio" — speech transcribed to text and back via separate models (roughly half his queries use this, via ChatGPT's mic button or the system-wide app SuperWhisper) — from "true audio," where speech is tokenized like text and handled natively inside the model, demonstrated live with ChatGPT's advanced voice mode switching between Yoda, pirate, and storytelling registers, and with Grok's more unfiltered persona modes. NotebookLM gets a similar demo generating a full spoken "podcast" discussion from an uploaded paper. On images, the same patch-based tokenization lets him upload nutrition labels, blood test screenshots, a toothpaste ingredient list, and a meme for explanation, generally transcribing to text first to verify the model read values correctly before asking follow-ups; DALL-E 3 handles image generation, and advanced voice's camera mode lets the model narrate what it sees live.

Quality-of-life features. ChatGPT's memory feature saves short text notes across conversations (e.g., a stated opinion on "peak Hollywood") that get silently prepended to future chats, which he says has made recommendations noticeably more personalized over time; custom instructions set a persistent tone and background (he uses his for a preferred formality level when practicing Korean); and custom GPTs bundle a reusable few-shot prompt, illustrated with three he built for Korean study — a vocabulary extractor, a detailed phrase-by-phrase translator, and an OCR-plus-translate tool for screenshotting hardcoded subtitles — each just a saved prompt with examples rather than anything architecturally new.

This is a general-audience walk through the entire pipeline behind a model like ChatGPT, in three stages — pretraining on internet text, supervised fine-tuning into a conversational assistant, and reinforcement learning that teaches the model to reason — using GPT-2, Llama 3, and DeepSeek-R1 as running examples, and closing with the cognitive quirks (hallucination, bad arithmetic, "Swiss cheese" failures) that fall out of how these systems actually compute. The destination is not a single trained model but a mental framework: understanding a chat reply as a lossy, statistical simulation of a human labeler, one token at a time, so the viewer can use these tools productively without over-trusting them.

Building the training data. The pipeline starts with Common Crawl, which has indexed roughly 2.7 billion web pages since 2007. Companies filter this raw HTML aggressively — blocking malware, spam, and adult sites; stripping markup down to plain text; running language classifiers (Hugging Face's FineWeb keeps only pages over 65% English, a deliberate choice that trades multilingual strength for English quality); deduplicating; and scrubbing personally identifiable information. What survives is surprisingly small given the internet's size: FineWeb totals about 44 terabytes, small enough to fit on a few consumer hard drives.

Tokenization. Before any of this text reaches a neural network, it has to become a sequence of symbols from a finite vocabulary. Raw UTF-8 bytes would give only 256 possible symbols and extremely long sequences, so production systems run byte-pair encoding, repeatedly merging the most common consecutive symbol pairs into new symbols. GPT-4 settles on about 100,277 tokens this way — a compression that trades a larger vocabulary for a shorter, cheaper-to-process sequence. Run through this process, FineWeb's 44 terabytes becomes about 15 trillion tokens.

Training the base model. A neural network — specifically a Transformer — takes a window of tokens (up to some maximum context length) and outputs a probability for every possible next token. Training means repeatedly sampling windows, checking the true next token against the model's guess, and nudging the network's parameters — billions of simple numeric "knobs" — so the correct token becomes slightly more likely. The output of this stage, after enough of these updates, is a base model: an internet document simulator with no persistent identity, no tools, and no ability to answer questions on demand — just a very good autocomplete. He demonstrates GPT-2's own numbers as a scale reference: 1.6 billion parameters, a 1,024-token context window, trained on about 100 billion tokens, originally costing roughly $40,000 in 2019. He reproduced it himself for around $600 on rented 8×H100 GPU hardware — and thinks $100 is achievable today — crediting better-curated data and much faster hardware and software, not a smarter algorithm. He also demonstrates that base models memorize high-quality, frequently-repeated sources almost verbatim (reciting a Wikipedia zebra article from memory) and that prompted cleverly — with few-shot examples, or a fake "conversation between a human and an AI assistant" transcript — a raw base model can already imitate an assistant, even though it was never trained to be one.

Supervised fine-tuning: manufacturing a personality. A base model is not useful for answering questions, so the next stage keeps the exact same training algorithm but swaps the dataset: instead of internet documents, the model now trains on curated conversations where human labelers write both the prompts and the "ideal" assistant responses, following lengthy company labeling instructions (OpenAI's InstructGPT paper is the source example, hiring contractors through Upwork and Scale AI). This stage is far cheaper than pretraining — hours instead of months — because the conversation dataset is much smaller. Modern practice increasingly uses LLMs themselves to help generate or edit this data (e.g., the largely-synthetic UltraChat dataset), but the process is still fundamentally anchored in human-written labeling instructions. The key reframe he offers: a ChatGPT answer is not oracle-like intelligence, but a neural network's statistical imitation of what a skilled human labeler, following those instructions, would have written — which is also why asking a model "what are you, who built you" produces confabulated answers (it will often just claim to be ChatGPT by OpenAI, because that phrase is common in its pretraining data) unless a company explicitly hardcodes self-identification into the fine-tuning data or a hidden system message.

Hallucination and its fixes. Because the model imitates the confident tone of its training conversations, when asked about someone it has never heard of it will invent a plausible-sounding but false biography rather than say "I don't know" — demonstrated live on the older Falcon-7B model inventing three different fake identities for a made-up name across three tries. Meta's Llama 3 paper describes the fix: interrogate the model with automatically generated questions about documents, use another model to judge whether its answers are correct, and where the model reliably fails, add training examples where the ideal response is an explicit "I don't know" — teaching the model to connect its own internal uncertainty to an honest refusal rather than a confident guess. A second, stronger mitigation is tool use: introducing special tokens that let the model emit a search query, pause generation, and have retrieved web text pasted into its context window, converting a "vague recollection" in the frozen parameters into a directly accessible fact in working memory — the same distinction, he notes, between something you memorized long ago and something you just looked up.

Tokens as a finite thinking budget. Because a Transformer performs a fixed, modest amount of computation per output token, no matter how many layers deep, a model can't correctly solve a hard problem if the label it was trained on crams the entire calculation into one token. Working through "Emily buys 3 apples and 2 oranges..." he shows that a labeled solution jumping straight to "$3" teaches the model to guess arithmetic instantly and it fails once the numbers get bigger (23 apples, 177 oranges), while a solution that writes out intermediate steps succeeds because each step only demands a small amount of per-token computation. The same constraint explains why models are weak at spelling and counting — they see tokens, not individual letters, so a task like printing every third character of "ubiquitous" or counting dots in a string routinely fails, and the "how many Rs in strawberry" meme traces to this same limitation, compounded by weak counting ability. His practical fix in both cases is the same: ask the model to write and run code rather than trust its "mental" arithmetic or letter-counting. He also flags an unresolved oddity — models frequently claim 9.11 is bigger than 9.9, and one research team traced this to neurons associated with Bible verse numbering lighting up and biasing the comparison, a reminder that these systems have a "Swiss cheese" pattern of capability: brilliant at Olympiad-level problems, yet capable of tripping on something trivial.

Reinforcement learning: the "practice problems" stage. He frames the three training stages as mirroring a textbook: pretraining is reading the exposition, SFT is studying worked examples with expert solutions, and RL is solving practice problems where only the final answer is known — the exact solution path is left for the model to discover through trial and error. Concretely, the model generates many candidate solutions to the same prompt, the correct ones are identified automatically against a known answer, and the model is trained to make more use of whatever token sequences led to correct answers — sequences discovered by the model itself, since a human labeler cannot reliably guess which reasoning steps are easy or hard for a given model's "cognition." Unlike the earlier two stages, which have been standard for years, RL is described as still early and non-standardized industry-wide — which is why DeepSeek's R1 paper, publishing its RL methodology openly, was such a significant event: it showed accuracy on math benchmarks climbing over thousands of training steps, alongside average response length growing dramatically as the model spontaneously developed backtracking behavior like "wait, that's not right, let me reevaluate" — a chain-of-thought strategy that emerged from optimization rather than being written by any human labeler. Given the same "Emily and the apples" prompt, R1 visibly double-checks its own arithmetic from a second angle before finalizing an answer, in contrast to a plain SFT model that just states one path to the answer. He draws an explicit parallel to DeepMind's AlphaGo, where reinforcement learning against self-play eventually surpassed models trained purely to imitate human experts, and produced "move 37" — a move so statistically unlikely for a human to play (about 1 in 10,000) that commentators initially thought it was a mistake, yet it proved brilliant; the open question he poses is what an equivalent breakthrough looks like in open-ended reasoning rather than a closed game like Go, and whether it might even involve the model finding a "language" for thinking that isn't English.

RLHF and why it's not quite RL. Verifiable domains like math have an automatic scoring function, but creative domains — writing a joke, a poem, a summary — don't. Reinforcement Learning from Human Feedback solves this with indirection: humans only need to rank a handful of model outputs (easier than writing an ideal response from scratch), and those rankings train a separate reward model to imitate human preference, which the language model can then be optimized against cheaply and at scale. The catch is that the reward model is itself a large neural network with exploitable blind spots: run RLHF too long and the policy discovers adversarial, nonsensical outputs — his example is a "joke" of literally "the the the the the" scoring near-perfectly — because reinforcement learning is extremely good at finding whatever quirk in the reward model produces a high score, whether or not it makes sense to a human. Unlike math RL, which can run for tens of thousands of steps because the scoring function (does the boxed answer match) can't be gamed, RLHF must be capped after a few hundred updates before it collapses into gibberish. His verdict: RLHF is real reinforcement learning technically, but not "RL with magic" — it's closer to a modest fine-tuning boost than an open-ended capability unlock, which is why GPT-4o-class models (mostly SFT plus a light layer of RLHF) behave differently from dedicated "thinking" models like o1, o3-mini, or R1, which have gone through genuine RL on verifiable problems.

What's next, and how to keep up. He expects near-term progress along several axes: native multimodality, where audio and images are tokenized the same way as text (spectrogram slices, image patches) and handled inside one model rather than stitched together from separate systems; longer-running, more autonomous "agents" that string together tasks over minutes or hours under human supervision, changing the operative metric to a human-to-agent ratio rather than human-to-robot; deeper integration and computer-use capability (citing ChatGPT's Operator as an early example); and open research problems like test-time training, since current models have no equivalent of a human's ability to actually learn from an ongoing experience rather than only holding it temporarily in a context window. For staying current, he points to the Chatbot Arena leaderboard (while cautioning it has become somewhat gameable and can rank models like Sonnet lower than actual usage would suggest), the "AI News" newsletter, and following knowledgeable people on X, alongside practical venues for using models directly: each provider's own site or app for proprietary frontier models, aggregators like Together AI for open-weight models such as DeepSeek and Llama, Hyperbolic specifically for base-model access, and LM Studio for running smaller, distilled, lower-precision models locally. His closing framing is the one he wants to stick: these are extraordinarily useful but fallible tools, riddled with "Swiss cheese" gaps even as they clear PhD-level benchmarks, and the correct posture is to use them for a first draft or for inspiration, verify their work, and keep ownership of the final result.

By the end of this build, a from-scratch GPT-2 124M — implemented in under 100 lines, trained on 8xA100 GPUs — matches OpenAI's original GPT-2 124M on validation loss and HellaSwag accuracy after roughly two hours on 10 billion tokens of FineWeb-Edu, and comes close to GPT-3's 124M checkpoint after an overnight, four-epoch, 40-billion-token run. Along the way, a chain of hardware- and algorithm-level optimizations cuts per-step time on an A100 from about 1,000ms to 93ms, an 11x speedup, at an estimated cloud cost of about $10 for the shorter run.

Loading the target model. Rather than starting from the original TensorFlow GPT-2 code released by OpenAI, the build starts from Hugging Face's PyTorch port, loading its state dict to inspect the real shapes: a 50,257×768 token embedding table (one row per BPE token, from the tokenizer covered in Karpathy's earlier tokenization series) and a 1,024×768 position embedding table. Plotting the position embeddings shows a jagged but broadly sinusoidal structure — evidence the vectors were learned from scratch rather than fixed as in the original 2017 Transformer, and a sign, from their raggedness, that this checkpoint isn't fully converged. Sampling five completions of "Hello, I'm a language model," from the Hugging Face pipeline gives a target: coherent, on-topic text to match once a custom implementation exists.

Rebuilding the architecture from scratch. GPT-2 departs from the original Transformer in two ways: it drops the encoder and cross-attention entirely (decoder-only), and it moves layer normalization to before each attention and MLP block instead of after, adding one more final layer norm before the classifier. The custom implementation mirrors Hugging Face's module names exactly (wte, wpe, h.0...h.11, ln_f) so weights can be copied over directly, and comes in at under 100 lines versus Hugging Face's roughly 2,000. Attention is described as a "reduce" — tokens exchange information by aggregating weighted values from earlier positions — while the MLP is a "map" applied identically to every token, making the Transformer a repeated map-reduce. The MLP uses GELU with the tanh approximation (GPT-2's original choice, made because the exact erf-based GELU was slow in TensorFlow at the time), preferred over ReLU because a smooth nonlinearity avoids the fully dead, zero-gradient neurons a flat ReLU tail can produce. Multi-head attention is implemented as one fused module using batched tensor reshaping — algorithmically identical to separate per-head modules, just faster — with a causal mask stopping any position attending to the future and a softmax normalizing attention weights to sum to one.

Loading real weights and validating generation. After copying OpenAI's released weights into the custom class (accounting for the fact that a few TensorFlow-derived weight matrices are transposed relative to PyTorch's convention), the custom model reproduces coherent completions, confirming the from-scratch architecture is correct — though its exact sampled tokens don't match Hugging Face's pipeline output bit-for-bit, a discrepancy Karpathy attributes to undocumented pipeline defaults (top-k 50 sampling among them) rather than a bug in the reimplementation. Switching to random initialization instead produces gibberish, as expected, and confirms the model is ready to train rather than merely replay OpenAI's weights.

A minimal training loop and sanity checks. Tokenizing the tiny Shakespeare dataset (about 1 million characters, roughly 338,000 GPT-2 tokens) and reshaping the token stream into (B, T) input batches, with targets simply the same sequence shifted by one position, gives the first working forward-backward-update loop using AdamW at a 3e-4 learning rate. Two sanity checks matter here: at initialization, the loss should be close to -ln(1/50257) ≈ 10.82, since an untrained model should assign roughly uniform probability across the vocabulary — the observed ~11 confirms a healthy random start; and the model should be able to overfit a single small batch to near-zero loss, which it does, confirming the optimization mechanics work before scaling up to real data.

Weight tying and initialization details. Inspecting the loaded state dict reveals that OpenAI's token embedding matrix and the final classifier's weight matrix are literally the same tensor — a weight-tying scheme from the original 2017 Transformer paper, motivated by the idea that semantically similar tokens should sit near each other in both the input embedding space and the output probability space. Pointing both layers at the same tensor saves about 40 million of the model's 124 million parameters, roughly 30%, and is expected to help a small model train more efficiently with fewer independent parameters to learn. For initialization, GPT-2's own source uses standard deviation 0.02 for weights and zero for biases (differing from PyTorch's default uniform bias init), values roughly consistent with the Xavier-style 1/√(fan-in) rule at typical GPT-2 channel counts. A second, less obvious detail scales every residual-path weight's standard deviation by an additional 1/√(2·n_layer): because each block adds its own contribution into the residual stream, repeatedly summing unscaled, unit-variance contributions inflates its variance — a toy example of 100 additions of standard-normal noise grows the running standard deviation to about 10 — so this correction keeps activation growth bounded regardless of depth.

Precision and compilation: from 1,000ms to 130ms per step. With batch size 16 and full GPT-2 sequence length 1,024, a plain float32 forward-backward step on an A100 costs about 1,000ms. Enabling TensorFloat-32 (TF32) — which truncates float32's mantissa from 23 to 10 bits inside the tensor-core matrix multiply only, transparently to the rest of the code — cuts this to about 333ms, a 3x gain rather than TF32's theoretical 8x, because the workload remains memory-bandwidth-bound: activations are still shuttled around in full float32 even though the multiply itself is faster. Switching to automatic mixed precision with bfloat16 (via torch.autocast, chosen over float16 because bfloat16 keeps float32's exponent range and needs no gradient scaler) brings this to about 300ms, converting activations like the logits to bf16 while parameters and operations like layer norm stay in float32. The largest single jump comes from torch.compile, which analyzes the whole model graph ahead of time rather than dispatching operations one by one from the Python interpreter, fusing elementwise sequences (GELU is the worked example) into single kernels that round-trip GPU high-bandwidth memory once instead of several times — dropping the step to about 130ms, a 2.3x gain from one line of code.

FlashAttention and vocabulary padding. torch.compile cannot find every fusion opportunity; the attention operation (torch.nn.functional.scaled_dot_product_attention, backed by FlashAttention) requires an algorithmic rewrite it won't discover on its own. FlashAttention performs more floating-point operations than the naive four-line attention implementation but is dramatically faster — the paper cites up to 7.6x — because it never materializes the full T×T attention matrix in high-bandwidth memory at all, instead computing softmax incrementally in an online, streaming fashion (a technique that traces back to a 2018 NVIDIA paper on online softmax normalization, four years before FlashAttention applied it). Swapping in FlashAttention drops the step to about 96ms. A final, smaller optimization pads GPT-2's awkward 50,257-token vocabulary up to 50,304 — a number divisible by 8, 16, 32, and 64 — because CUDA kernels operate in fixed-size block tiles and an input that doesn't divide evenly forces slow boundary-handling code paths; despite the added compute from the extra fake tokens (which the model simply learns to assign near-zero probability), this padding brings the step time down to about 93ms, an roughly 11x total speedup from the 1,000ms starting point.

Matching GPT-3's optimization hyperparameters. Because GPT-2's own paper is vague on optimizer details, hyperparameters are pulled from GPT-3's paper instead, given the architectures are otherwise very similar. This means AdamW with betas (0.9, 0.95) and epsilon 1e-8 in place of PyTorch's defaults, global gradient-norm clipping at 1.0 to guard against destabilizing loss spikes, and a cosine learning-rate schedule with linear warmup — rising to a peak of 6e-4 for a 124M-parameter model, then decaying to 10% of that peak, implemented as under a dozen lines rather than PyTorch's built-in scheduler so every step of the math stays visible. Weight decay of 0.1 is applied only to two-dimensional parameters (matrix weights and embeddings), leaving one-dimensional biases and layer-norm parameters undecayed, and a fused AdamW kernel (available in newer PyTorch, detected via inspect.signature) shaves the step time slightly further by updating all parameter tensors in one kernel launch instead of iterating over them individually.

Gradient accumulation, and the bug in getting it right. GPT-3's stated 0.5-million-token batch size doesn't fit in one GPU's memory, so gradient accumulation simulates it: multiple forward-backward passes accumulate gradients before a single optimizer step. A toy example with a 4-example batch versus four size-1 gradient-accumulation steps demonstrates the trap: because the loss functions used (MSE, and later cross-entropy) default to averaging over the batch, naively accumulating per-example losses via repeated .backward() calls sums rather than averages the contributions, producing gradients too large by the accumulation factor; the fix is dividing each micro-batch's loss by the number of accumulation steps before calling backward, recovering an identical result to true large-batch training.

Scaling to 8 GPUs with DistributedDataParallel. Launched via torchrun, eight copies of the training script run in parallel, each pinned to one GPU and identified by a DDP rank used to stride each process through a distinct slice of the data loader so no two GPUs see the same batch. DDP keeps the forward pass unchanged but, after the backward pass, averages gradients across all eight ranks via an all-reduce, so every rank ends the step with identical, batch-averaged gradients; to avoid wasteful synchronization on every micro-step during gradient accumulation, Karpathy directly toggles DDP's internal require_backward_grad_sync flag so the all-reduce fires only on each macro-step's final micro-step, rather than using PyTorch's documented (but, in his view, uglier) no_sync context-manager pattern. The accumulated loss value itself also needs an explicit dist.all_reduce average across ranks before the master process logs it, since otherwise each process would only report the loss it locally observed. With 8 A100s, gradient accumulation drops to 4 steps and throughput reaches about 1.5 million tokens per second.

Data: FineWeb-Edu, and evaluation with HellaSwag. GPT-2 and GPT-3 both trained on undisclosed, Reddit- and Common-Crawl-derived datasets, so the build substitutes a 10-billion-token sample of Hugging Face's FineWeb-Edu, a Common Crawl subset filtered for educational content (using Llama 3 70B as an automatic quality judge), tokenized into 100 shards of 100 million tokens apiece. Alongside a held-out validation shard, the model is periodically scored on HellaSwag, a 2019 multiple-choice sentence-completion benchmark where wrong answers are adversarially generated to fool language models while remaining easy for humans (95% human accuracy versus 48% for the best models of that year); since a 124M model can't natively perform multiple-choice reasoning, each candidate ending is instead scored by its average per-token loss under the model, with the lowest-loss completion taken as the prediction, and GPT-2 124M is measured at 29.5% (versus 25% random chance) as the baseline to beat.

Results and open caveats. Training for one epoch (about 19,073 steps, 10 billion tokens, roughly two hours) produces a validation loss below OpenAI's GPT-2 124M checkpoint's own score on this validation set, and a HellaSwag accuracy surpassing GPT-2 124M's 29.5% baseline — with 10x fewer training tokens than GPT-2's original 100-billion-token budget, tentatively credited to FineWeb-Edu's narrower, English-only, higher-quality domain and to broader post-2019 gains in data curation and filtering. An overnight extension to four epochs (about 40 billion tokens) pushes HellaSwag to 33.24%, nearly reaching GPT-3 124M's own reported score despite its 300-billion-token budget. Two rough edges stay unresolved: a periodic, sawtooth pattern in the loss curve, suspected to come from FineWeb-Edu shards not being shuffled or permuted across epochs, and an unexplained error that forces torch.compile off during generation and HellaSwag evaluation. The lecture closes by pointing to llm.c, a parallel CUDA/C reimplementation that reproduces identical losses and norms step-for-step while running faster than this PyTorch version, and by noting that everything built here is pretraining only — a conversational model, as in ChatGPT, would need a further supervised fine-tuning pass on conversational data, not covered here.

By the end of this build he has implemented, from scratch in Python, a working byte-pair-encoding tokenizer capable of reproducing GPT-4's own tokenization behavior — training a vocabulary, encoding text to tokens, and decoding back — and used that implementation to explain a long list of otherwise-mysterious LLM failures, from bad arithmetic to a glitch token that makes models swear at users. The throughline is that tokenization is a completely separate preprocessing stage from the language model itself, with its own training set and algorithm, and that a surprising fraction of what looks like a model or architecture problem actually traces back to this stage.

Why not just feed in raw bytes or characters. His earlier character-level tokenizer (from the "build GPT from scratch" lecture) used a vocabulary of only 65 symbols, one per character seen in a small dataset — simple, but impractical at scale. Text in Python is a sequence of Unicode code points, of which roughly 150,000 exist, but using code points directly is unattractive both because the standard keeps changing and because encoding to raw UTF-8 bytes gives a vocabulary of only 256 symbols, which would stretch every document into extremely long byte sequences that a Transformer's finite context window and attention cost can't accommodate efficiently.

The byte-pair encoding algorithm. The fix is to compress those byte sequences by repeatedly finding the most frequently occurring adjacent pair of tokens and minting a new token to represent it, replacing every occurrence — for example, merging the common pair "e" + " " (space) into a single new token once it's identified as the single most frequent pair in a sample paragraph. Repeating this some chosen number of times (a hyperparameter — GPT-4 does enough merges to reach a vocabulary near 100,277) builds up a "forest" of merges on top of the 256 base byte tokens, and this merge table, once built, is all that's needed to both encode new text into tokens and decode tokens back into text. He demonstrates this concretely: 20 merges on a ~24,000-byte blog excerpt compress it to about 19,000 tokens, a 1.27x ratio that improves further with more merges. Critically, vocabulary size can't grow without limit — a bigger vocabulary means a bigger embedding table and a bigger final classification layer, more parameters that individual rare tokens will under-train, and a risk of cramming too much meaning into single tokens for the network's fixed per-token computation to unpack.

GPT-2's regex-based pre-splitting. OpenAI didn't apply plain BPE to raw text; GPT-2's released code (encoder.py) first splits text using an elaborate regex that separates letters from numbers from punctuation from whitespace, so that merges can never cross those boundaries — preventing nonsensical joint tokens like "dog!" merging as a unit with "dog?" while still keeping the two semantically identical. He steps through the regex clause by clause, showing quirks like inconsistent handling of apostrophes (a Unicode right-single-quote isn't matched the way a plain ASCII apostrophe is) and case-sensitivity bugs GPT-2's own documentation admits to. GPT-4's tokenizer (used via the tiktoken library, vocabulary roughly double GPT-2's ~50k) revised this pattern to be case-insensitive and, notably, to group runs of whitespace into single tokens rather than leaving each space isolated — a change he shows cuts token counts for indented Python code roughly in half, and identifies as a real contributor to GPT-4's markedly better code handling, separate from any model or training improvement.

Special tokens and model surgery. Beyond byte-derived tokens, tokenizers reserve dedicated IDs for structural markers — GPT-2's single <|endoftext|> token delimits documents during pretraining, while chat models add a family of tokens (<|im_start|>, <|im_end|>, and others) to mark turns between user and assistant. Adding a new special token to a pretrained model requires literal "model surgery": extending the embedding table with a new randomly-initialized row and widening the final output layer by one, which is routine practice when moving a base model toward a fine-tuned chat or tool-using model.

SentencePiece as an alternative. Used by Llama and Mistral, SentencePiece differs fundamentally from tiktoken by running BPE merges directly on Unicode code points rather than on UTF-8 bytes; any code point too rare to have a token gets mapped to a fallback — either a generic unknown token, or, if "byte fallback" is enabled (as in Llama 2), decomposed into UTF-8 bytes and represented by dedicated byte tokens instead. He walks through SentencePiece's large, historically-accumulated configuration surface (sentence-splitting assumptions, normalization options, a mandatory unknown token, a "dummy prefix" space added to make a word look the same whether or not it starts a sentence) and judges it considerably less clean than tiktoken's approach, though it remains widely used because — unlike tiktoken — it supports training a vocabulary, not just running inference with one.

Multimodal tokenization and vocabulary extension. He notes briefly that the same Transformer architecture is increasingly used across modalities simply by inventing new tokenizers — chunking images into patches (citing an early image-tokenization paper and OpenAI's Sora, which represents video as "visual patches" analogous to text tokens) rather than by changing the model itself. Separately, techniques like "gist tokens" show vocabularies can be extended purely to compress long prompts: new tokens are trained by distillation, with the rest of the model frozen, so a handful of learned tokens can later stand in for an entire long prompt with near-identical model behavior.

Diagnosing real failures through tokenization. He closes by tracing a list of well-known LLM quirks back to this stage. Long, high-frequency tokens like .DefaultCellStyle bundle many characters into one indivisible unit, so a model asked how many "L"s it contains gets the answer wrong — it never sees individual letters, only the whole chunk — while asking it to first print the characters separated by spaces and then reverse them works, because that breaks the token apart. Arithmetic suffers similarly: digit groupings into tokens are arbitrary and inconsistent (a four-digit number might be one, two, three, or four tokens depending on what merged), disrupting the character-level carrying logic addition needs — which is why Llama 2 explicitly splits all digits during tokenization. Non-English text tokenizes far less efficiently since the tokenizer's own training data skews English — a Korean greeting can take three tokens versus one for its English equivalent, a roughly 3x expansion that effectively shrinks the usable context window for other languages. Oddities like a completion API breaking on trailing whitespace, or choking on the literal string "<|endoftext|>", stem from the mismatch between how partial or special tokens are represented internally versus how they appear in raw input. The most vivid example is "SolidGoldMagikarp": a Reddit username that appeared often enough in the tokenizer's own training corpus to earn a dedicated token, but never occurred in the language model's actual training data — leaving that embedding row permanently untrained and effectively random, so invoking the token later triggers undefined behavior (evasion, hallucination, even insults), which he compares to reading uninitialized memory in a C program. He ends with a practical recommendation: reuse GPT-4's existing tokenizer and tiktoken library where possible, and be wary of SentencePiece's many easily-miscalibrated settings if training a custom vocabulary is unavoidable.

This is a conceptual talk rather than a coding session, building toward one closing image: a large language model as the kernel process of an emerging operating system — reading and writing text, browsing, running code, seeing and generating images — with a new attack surface attached.

What a model is, and how it becomes an assistant. A model like Llama 2 70B is really just two files: parameters and a few hundred lines of run code, small enough to run offline on a laptop. The parameters are expensive to obtain — Meta compressed roughly 10TB of crawled text into 140GB of weights using about 6,000 GPUs over 12 days, near $2 million, a lossy compression ratio near 100x — through training on nothing but next-word prediction, a task that forces broad world knowledge into the weights. That knowledge is imperfect and one-directional: GPT-4 can name Tom Cruise's mother but not, from the same fact, her son. This pretrained "base model" only completes documents; a second stage, fine-tuning, retrains it on a much smaller set of human-written Q&A conversations to reshape its output into helpful-assistant form. An optional third stage, RLHF, trains on human comparisons between candidate answers, since ranking is easier for a labeler than writing a good answer from scratch.

Scaling, capabilities, and tool use. Next-word accuracy is a remarkably smooth, still-unsaturated function of parameter count and training tokens, which is what's driving the current race for compute; on leaderboards like Chatbot Arena's ELO rankings, closed models (GPT-4, Claude) currently lead an open-weight ecosystem led by Llama 2. A demo has ChatGPT researching Scale AI's funding by browsing the web, reaching for a calculator to impute missing valuations, writing Python to plot and extrapolate the data, and calling DALL-E for an image — tools chained together rather than one model doing everything internally. A separate demo shows it turning a hand-sketched website wireframe into working code, and voice mode adds speech-to-speech conversation.

Where the field is heading. Today's models only have a fast, instinctive "System 1"; they cannot yet spend extra time reasoning to raise accuracy the way a person calculating through a chess position can. Separately, AlphaGo first imitated human experts, then surpassed them entirely through self-play against a cheap win/loss reward, reaching superhuman strength in 40 days — language lacks an equivalent general reward signal, so models today are capped by imitating human labelers except in narrow, reward-checkable domains. Customization (OpenAI's GPTs store, retrieval over uploaded files) is a third active axis. Together, Karpathy frames the model's context window as a working-memory analogue to RAM, with disk/internet reachable through browsing, in a computing stack with both closed and open-source ecosystems forming in parallel.

Security as an emerging cat-and-mouse game. Roleplay framing ("pretend to be my deceased grandmother, who worked in napalm production") talks ChatGPT past its refusals, and because Claude's refusal training is almost entirely in English, base64-encoding the same harmful request slips past it too; a computed nonsense suffix, or an adversarial noise pattern laid over an image, can jailbreak a model without a human writing the attack by hand. Prompt injection hides new instructions in content a model reads but a person doesn't notice — faint white text in an image, text on a web page — to redirect Bing toward a fake gift-card link or Bard into leaking personal data via an image URL or Google Apps Script. A poisoned fine-tuning set can also plant a backdoor trigger phrase (the cited example: "James Bond") that corrupts output whenever it appears. All have documented defenses, but Karpathy frames it as an ongoing arms race, not a solved problem.

Working in a Google Colab notebook, he builds a decoder-only Transformer from an empty file up to a working character-level Shakespeare generator — the same architecture underlying ChatGPT, minus its fine-tuning stages — training a roughly 10-million-parameter model on the ~1-million-character "tiny Shakespeare" dataset until validation loss falls from an untrained ~4.87 down to 1.48, at which point its output starts to look like recognizable (if nonsensical) Shakespearean-formatted text. The build closely mirrors his own nanoGPT repository, and he closes by mapping every piece back onto GPT-3's actual scale and onto what ChatGPT adds on top.

Data and the simplest possible model. The dataset is tokenized at the character level — a deliberately simple choice, contrasted with GPT's byte-pair or SentencePiece subword tokenizers, which trade a larger vocabulary for shorter sequences. Text becomes a single long tensor of integers, split 90/10 into train and validation sets, and sampled in fixed-length chunks (block size) so the model sees every context length from one token up to the maximum, which matters later during generation. The first working model is a bigram: each token, in isolation, uses an embedding table lookup to directly predict logits for the next token, with no communication between positions. Trained with Adam, this baseline's loss falls from about 4.7 to roughly 2.5 — proof the optimization loop works, but the output is gibberish because each character predicts the next based only on its own identity.

The mathematical trick behind attention. To let earlier tokens influence a later prediction without expensive looping, he shows that multiplying a token sequence by a row-normalized lower-triangular matrix computes a running causal average in one matrix multiplication — averaging in a token's own vector with all preceding ones while never looking ahead. Expressed as a masked softmax (zeroing future positions to negative infinity before normalizing), this same operation becomes the scaffold for self-attention once the uniform triangular weights are replaced by learned, data-dependent ones.

Self-attention itself. Each token emits three vectors via separate linear projections: a query ("what am I looking for"), a key ("what do I contain"), and a value (what actually gets aggregated once another token finds it interesting). Attention weights come from the dot product of every query against every key, masked to block future positions, then softmaxed and used to weight-sum the values — meaning tokens dynamically decide, based on their own content, how much to attend to each earlier token, rather than always averaging uniformly. A critical detail: the raw dot products are divided by the square root of the head size before the softmax, because otherwise their variance grows too large at initialization, driving softmax toward near one-hot outputs that let each token learn from effectively only a single other token. He's careful to frame attention as a general communication mechanism over a directed graph, not something intrinsically tied to language — the causal, future-blind mask is specifically what makes it a decoder rather than the fully-connected encoder used in tasks like sentiment classification, and cross-attention (queries from one source, keys/values from another) is what lets an encoder-decoder system like the original machine-translation Transformer condition generation on a separately-encoded input sequence.

Multi-head attention and computation. Running several smaller attention heads in parallel and concatenating their outputs — analogous to grouped convolutions — lets different heads specialize in different relationships (e.g., a vowel token searching specifically for nearby consonants) rather than forcing one attention pattern to do everything; this dropped validation loss from 2.4 to 2.28. Because attention only lets tokens communicate, without giving them time to "think" about what they gathered, a small per-token feedforward MLP (linear → ReLU → linear) is added after each attention step, further improving loss to 2.24.

Why depth needs residual connections and LayerNorm. Simply stacking multiple communicate-then-compute blocks made the network start to degrade rather than improve, a classic deep-network optimization problem. The fix, borrowed directly from the 2015 ResNet paper, is residual (skip) connections: each block forks off the main pathway, transforms, and adds its output back in, so gradients have an unimpeded "highway" back to the input during backpropagation, with each block's contribution initialized near zero and growing in over training. Paired with LayerNorm (which, unlike BatchNorm, normalizes each token's own feature vector rather than across the batch) applied before rather than after each sub-layer — a deliberate deviation from the original paper's post-norm design, adopted because it works better in practice — this let him stack six full blocks and reach a validation loss of 2.06, with dropout added to control overfitting as the model grew.

Scaling up and comparison to GPT-3. With block size raised to 256, embedding dimension to 384, six attention heads, six layers, and dropout at 0.2 — about 15 minutes of training on an A100 GPU — validation loss reaches 1.48, and generated text starts to visually resemble the structure of the source play. He then sets this ~10-million-parameter model, trained on roughly 300,000 GPT-vocabulary-equivalent tokens, against GPT-3's actual specs: architecturally near-identical hyperparameter choices, but scaled to 175 billion parameters trained on 300 billion tokens — a roughly millionfold increase, requiring thousands of coordinated GPUs rather than one.

Why pretraining alone isn't ChatGPT. A model trained only this way is a "document completer": given a question, it may just continue with more questions or drift into unrelated text, because it has only learned to imitate whatever an internet document containing that text tends to look like. Turning that into an assistant requires a separate, mostly non-public alignment pipeline he did not implement here: supervised fine-tuning on a smaller set of curated question-and-answer-formatted conversations, followed by training a reward model from human rankings of candidate responses, followed by using that reward model to further tune the policy via PPO reinforcement learning — the process OpenAI describes in its ChatGPT blog post, and which nanoGPT itself only covers up through the pretraining stage.

Starting from the part-3 character-level MLP (three characters of context feeding one hidden layer, validation loss 2.10), he restructures it into a WaveNet-style hierarchical network — progressively fusing pairs of characters across multiple layers instead of squashing all context into one layer at once — and lands, after fixing a BatchNorm bug and scaling up, at a validation loss of 1.993 on the same names dataset, while also converting the codebase to a small PyTorch-like module library (Embedding, Flatten, Sequential) reusable going forward.

PyTorch-ifying the codebase. Before touching the architecture, he refactors the ad hoc layer list into proper module classes — Embedding and Flatten alongside the existing Linear, BatchNorm1d, and Tanh — plus a Sequential container that chains them, mirroring torch.nn's own API. This isn't cosmetic: the forward pass collapses to a single call through the model, exposing a real bug where forgetting to leave BatchNorm in evaluation mode (rather than training mode) caused a single-example forward pass to compute a variance over one number, silently corrupting output — a reminder that BatchNorm's train/eval state is an easy source of hidden bugs.

Baseline: just widening the flat network. Simply increasing context from 3 to 8 characters in the existing one-hidden-layer architecture drops validation loss from 2.10 to 2.02, with no structural change — establishing a baseline against which the hierarchical redesign has to earn its keep.

Building the hierarchical (WaveNet-style) network. Rather than flattening all 8 characters into one long vector immediately, he fuses them progressively: a FlattenConsecutive layer groups every 2 adjacent character embeddings into a single unit, which a linear/BatchNorm/Tanh block processes; this repeats, so 8 characters become 4 pairs, then 2 groups of 4, before a final layer produces the prediction — matching WaveNet's tree-like, dilated causal convolution structure, though implemented purely by reshaping tensors rather than by using real convolutions. He shows PyTorch's matrix multiply already supports this: it operates only on a tensor's last dimension, treating any number of leading dimensions as batch dimensions, so grouping character-pairs into a new "batch-like" dimension is just a view/reshape, not a new operation.

The 3D BatchNorm bug. Feeding the now-3D intermediate tensors (batch × groups × channels) into a BatchNorm1D layer written for 2D input caused it to silently compute separate running statistics per group instead of per channel — reducing only over dimension 0 instead of over both the batch and group dimensions. The fix (reducing jointly over dimensions 0 and 1 when the input is 3D) recovers a proper per-channel normalization and gives a small but real improvement, from 2.029 to 2.022 — and he notes this convention deliberately departs from PyTorch's own BatchNorm1d, which places the channel dimension in the middle rather than last.

Results and what convolutions actually buy you. With the corrected hierarchical architecture and its parameter budget matched to the flat baseline, results are initially a wash (~2.02 either way), but scaling embedding and hidden sizes up (to about 76,000 parameters) pushes validation loss to 1.993. He closes by clarifying that real WaveNet-style convolutions wouldn't change what's computed here at all — they're purely an efficiency mechanism, sliding the same shared filters across a sequence inside fast CUDA kernels and reusing intermediate values shared between overlapping positions, rather than recomputing the whole hierarchy independently at every character position as his implementation does.

Rather than adding a new architecture, this lecture strips one away: he takes the two-layer MLP-plus-BatchNorm character model from the previous lecture and rips out loss.backward(), replacing PyTorch's autograd with a fully hand-derived backward pass, tensor by tensor, until the same model trains to the same loss using nothing but manually computed gradients. The point isn't the result — it's that backpropagation is what he calls a "leaky abstraction," and skipping straight to autograd hides exactly the kind of bugs (a real-world example: a codebase that clips the loss instead of the gradient, silently zeroing outliers' influence) that only someone who's derived it by hand will catch.

Historical framing. He opens by noting that hand-written backward passes were standard practice barely a decade ago — showing his own 2010 Matlab code for training restricted Boltzmann machines and 2014 NumPy code for an image-sentence embedding model, both with manually implemented gradients checked against numerical estimates — before autograd engines like PyTorch's made it unnecessary, and argues that understanding the mechanism is still what separates someone who can debug a broken network from someone who can only rerun it.

Exercise 1: full atomic backpropagation. Working through the forward pass broken into its smallest pieces — log-probabilities, counts, exponentials, matrix multiplies, tanh, and every BatchNorm sub-step — he derives each gradient by hand and checks it against PyTorch's own .grad values. Two recurring principles do most of the work: local derivatives compose via the chain rule exactly as in his earlier micrograd project, and there is a duality between forward-pass operations and backward-pass shapes — summing in the forward pass (as when a bias vector or BatchNorm parameter gets broadcast, i.e., replicated, across many rows) means broadcasting the gradient back to that same shape in reverse, and vice versa; a variable reused multiple times (like a shared embedding row) must have its gradients from every use added together. Matrix multiplication derivatives, rather than being memorized, are derived by writing out a tiny 2×2 example element-by-element and pattern-matching the resulting shapes, arriving at the standard dA = dOut · Bᵀ, dB = Aᵀ · dOut results without invoking matrix calculus formalism.

Exercise 2: the analytical shortcut for cross-entropy. Rather than backpropagating through the six-plus intermediate steps of softmax and negative-log-likelihood individually, he derives on paper that the gradient of the combined loss with respect to the logits collapses to something remarkably clean: the softmax probabilities, with 1 subtracted at the position of the correct label, all divided by batch size. Visualizing this gradient row by row, he offers a physical intuition — think of it as a tug-of-war where the correct answer's probability gets pulled up and every incorrect answer's probability gets pulled down, with the two forces in each row exactly balancing (summing to zero), and the strength of each pull directly proportional to how wrong the current prediction is; a perfectly confident correct prediction produces no gradient at all.

Exercise 3: the analytical shortcut for BatchNorm. He repeats the same collapsing exercise for BatchNorm, which is considerably harder because both the mean and variance are computed from, and feed back into, every element of the batch, creating multiple gradient paths that must be summed via the chain rule before they cancel and simplify into one manageable closed-form expression — verified by matching PyTorch's own backward pass to within floating-point precision. Along the way he flags what he considers a real bug: the original BatchNorm paper (and PyTorch's own implementation) use the biased variance estimator (dividing by n) during training but switch to the unbiased, Bessel-corrected version (dividing by n−1) for the running statistics used at inference — a train/test inconsistency with no clear justification, which he resolves in his own implementation by using the unbiased estimator consistently in both places.

Exercise 4: putting it together. With every gradient now derivable from a compact, roughly 20-line backward pass — rather than the sprawling atomic version from Exercise 1 — he retrains the full network end to end using only these manually computed gradients, with loss.backward() and PyTorch's autograd entirely absent, reaching essentially the same loss and generating the same quality of name-like samples as before. The exercise's real deliverable isn't a better model; it's confidence, built layer by layer, that nothing about how these networks learn is mysterious — only, in BatchNorm's case, a bit more algebraically involved than the rest.

Staying with the same character-level MLP from the previous lecture rather than moving to recurrent networks as planned, he spends the session diagnosing and fixing exactly what can go wrong inside a neural network's activations and gradients at initialization — pushing validation loss from 2.17 down to about 2.10 through nothing but better initialization and normalization — because, as he argues, these same failure modes become far more dangerous once networks get much deeper, which is the whole reason to understand them now.

The hockey-stick loss. Fresh off training, he notices the very first loss recorded is 27, wildly higher than the ~3.29 a 27-way uniform guess should cost (negative log of 1/27). The cause: default initialization produces logits with large, arbitrary magnitudes, so the softmax comes out confidently wrong for most examples rather than diffuse; the first chunk of training is then spent simply shrinking those logits back toward zero, visible as a sharp early drop in the loss curve before real learning starts. The fix is to initialize the final layer's weights (and biases) near zero, which immediately produces the expected initial loss, removes the hockey-stick shape, and — because training time is no longer wasted correcting an artificial problem — measurably improves the final loss.

Saturated tanh units and dead neurons. A histogram of the hidden layer's activations reveals most values pinned at exactly +1 or -1 — tanh fully saturated. This matters because tanh's backward-pass gradient is (1 − output²): an output near ±1 multiplies the incoming gradient by nearly zero, effectively cutting off learning for that unit on that example. Checking column-by-column for neurons where every example saturates reveals the more serious failure mode: a "dead neuron" that never receives a gradient again and is permanently stuck — a risk he notes is shared by sigmoid (also a squashing function) and ReLU (a dead ReLU, unable to recover once knocked into producing only negative pre-activations, sometimes triggered by too-high a learning rate). The fix mirrors the logit fix: shrink the weights and biases feeding into the hidden layer so pre-activations land in a reasonable range, eliminating the saturated tail and further improving loss.

Principled initialization. Rather than hand-picking scale factors by trial and error, he derives the general rule from Kaiming/He initialization: to keep a layer's output variance equal to its input's, weights should be drawn with standard deviation equal to a gain divided by the square root of fan-in. The gain compensates for how much a given nonlinearity compresses its input distribution — 1 for a purely linear layer, but roughly 5/3 for tanh, whose squashing otherwise causes activation variance to shrink toward zero as it passes through successive layers (demonstrated directly: gain too low and deeper layers' activations collapse toward zero; gain too high and they oversaturate). He's candid that this precise tuning mattered enormously before several later innovations — residual connections, various normalization layers, and better optimizers like Adam — made networks considerably more forgiving of imperfect initialization.

Batch normalization. The core insight, introduced by a 2015 Google paper, is disarmingly direct: since what's wanted is roughly Gaussian pre-activations at every layer, why not just standardize them directly — subtract the batch's mean, divide by its standard deviation — since that operation is fully differentiable? To avoid permanently forcing a fixed distribution, this is paired with a learnable gain and bias so the network can still shift or reshape the distribution as training demands. He implements it as a drop-in layer placed after linear layers and before nonlinearities, demonstrating that it makes networks dramatically more robust to the exact scale of the preceding weights — even removing fan-in scaling entirely still trains reasonably once BatchNorm is present, though the learning rate may need retuning since BatchNorm's own backward pass changes how gradient scale relates to the weights feeding into it.

The catch is a real one: normalizing over the batch means an example's hidden state is no longer a pure function of that example — it now depends on whichever other examples happened to land in the same batch, introducing "jitter" that acts as an accidental regularizer but is also, he says, a persistent source of subtle bugs, which is why later techniques (layer, instance, and group normalization) were developed to decouple examples again. A related practical wrinkle: since BatchNorm re-centers its input, any bias term in the preceding linear layer is mathematically canceled out and wastes a parameter, so linear layers immediately before a BatchNorm should disable their bias. For inference on a single example, where batch statistics don't exist, the layer instead uses a running mean and variance accumulated via exponential moving average during training — the same approach used in PyTorch's own BatchNorm1d, whose parameters (momentum, epsilon, affine, track_running_stats) he walks through directly, noting that overly large momentum can be unstable for small batch sizes.

Diagnostics for a healthy network. Beyond the fixes, he introduces the habit of actively monitoring three things during training: histograms of forward-pass activations and backward-pass gradients across layers, watching for saturation or asymmetric shrink/growth with depth, and — most usefully — the ratio of each parameter's update size to its own data scale, plotted on a log scale over training. His rule of thumb is that this ratio should hover around 10⁻³; consistently far below suggests the learning rate is too low, far above suggests it's too high, and BatchNorm-equipped networks are shown to be considerably more forgiving of these settings than the raw linear-tanh stack was. He closes by noting these gains didn't actually beat the model's previous best loss — because the real bottleneck at this point is a context window of only three characters, not optimization — foreshadowing the shift to recurrent architectures and, eventually, Transformers.

Moving past the bigram model's fundamental limit — its lookup table grows exponentially with context length, making anything beyond a single prior character intractable — this lecture implements a multi-layer perceptron character-level language model following Bengio et al. 2003, ending with a network that takes three characters of context, embeds them, and predicts the fourth, reaching a validation loss of about 2.17 (versus 2.45 for the bigram model) and producing noticeably more name-like samples like "ham," "joes," and "lela."

The architecture. Each of 27 possible characters is mapped to a point in a low-dimensional embedding space via a lookup table C, initialized randomly and tuned through backpropagation alongside the rest of the network. He demonstrates that indexing into this table is mathematically equivalent to multiplying a one-hot vector by C — framing the embedding step as just a linear layer with no nonlinearity — before settling on the much faster direct-indexing approach. Three character embeddings are concatenated (efficiently, via .view() rather than torch.cat, since reshaping a tensor's view requires no new memory) and fed through a hidden layer with a tanh nonlinearity, then a final linear layer producing 27 logits. The motivating idea from the paper: since embeddings place similar characters or words near each other in this learned space, the model can generalize to combinations it never saw in training — if it learns "a" and "the" behave similarly, it can transfer what it learned about "the dog was running" to "a dog was running" even without that exact phrase in the data.

Practical training mechanics. He switches the hand-rolled softmax-and-negative-log-likelihood computation to PyTorch's F.cross_entropy, noting it's not just more convenient but meaningfully better: it fuses operations into a faster forward and a mathematically simpler backward pass, and critically avoids numerical overflow by internally subtracting the batch's maximum logit before exponentiating — since softmax output is invariant to a constant offset, this keeps very large logits from producing inf. Training then moves to mini-batches of 32 random examples rather than the full 228,000, trading a noisier, approximate gradient for dramatically more update steps in the same time — which turns out to converge faster overall since gradient direction, even if imperfect, is still useful.

Finding a learning rate. Rather than guessing, he sweeps candidate learning rates exponentially from 0.001 to 1 across a short run and plots the resulting loss against the learning rate's exponent, producing a characteristic valley: too low and the loss barely moves, too high and training becomes unstable. The middle of that valley — around 0.1 — gives a defensible starting point, later followed by a manual 10x decay once the loss plateaus, a two-stage schedule he's candid is "janky" compared to production practice but effective enough here.

Diagnosing underfitting versus overfitting. Splitting the data 80/10/10 into train, dev (validation), and test sets, he shows that when training and dev loss track closely (~2.3 each) but remain fairly high, the model is underfitting — too small to be memorizing, let alone generalizing well — and needs more capacity rather than regularization. Tripling the hidden layer's neuron count barely moved the needle, but increasing the character embedding dimensionality from 2 to 10 (with the hidden layer trimmed back to 200 units) dropped the loss meaningfully further, revealing that the 2-dimensional embedding space, not the hidden layer, was the actual bottleneck restricting how much distinct information the network could represent per character. Visualizing the trained 2D embeddings before scaling them up makes this concrete: vowels cluster tightly together, ordinary consonants occupy a shared middle region, and outliers like "q" and the special "." token sit far off on their own — visible evidence that the space is encoding real structure rather than sitting at its random initialization. The chapter closes by explicitly flagging the knobs left untuned — embedding size, hidden layer width, context length, batch size, and learning rate schedule — as an open invitation to keep improving past the 2.17 validation loss reached here.

This lecture builds the first and simplest member of the makemore family — a bigram, character-level name generator — twice over: once by directly counting character-pair frequencies, and once by training a single-layer neural network with gradient descent, arriving at the identical ~2.45 loss and identical output samples both ways. The destination isn't a strong model (bigram statistics are, in his words, "just really terrible" — producing garbled near-names rather than good ones) but a fully worked bridge between classical counting-based statistical modeling and the gradient-based neural framework that every later lecture in the series builds on.

Counting bigrams. Starting from a 32,000-name dataset, he treats each name as a sequence of bigrams — consecutive character pairs — wrapping every word with a special start/end token (later unified into a single "." token) so the model also learns which characters are likely to begin or end a name. Every one of the roughly 27x27 possible pairs (26 letters plus the special token) gets a count, tallied into a PyTorch tensor and visualized as a heatmap, revealing structure like "N" being an extremely common ending character.

From counts to a generative model. Normalizing each row of the count matrix into a probability distribution turns raw statistics into something samplable: starting from the special start token, torch.multinomial draws a next character according to that row's probabilities, then repeats using the new character's row, until the end token is drawn. This produces recognizably name-adjacent gibberish ("Keanu," "O'Reilly"-ish fragments), which he confirms is genuinely better than a uniform baseline, even though it's still clearly a weak model since it only ever looks one character back.

Measuring model quality and its sharp edges. To turn "how good is this model" into a single number, he introduces the likelihood (the product of the probabilities the model assigns to every actual next character in the training data) and its practical cousins: log-likelihood (summing logs instead of multiplying tiny numbers) and negative log-likelihood, averaged, which becomes the loss to minimize — lower is better, zero is perfect. A sharp edge appears immediately: querying the model on a plausible but unseen bigram (a name ending in "...jq") returns exactly zero probability, and log(0) is negative infinity, breaking the loss entirely. The fix, "model smoothing," adds a small fake count to every bigram before normalizing, guaranteeing no probability is ever exactly zero — trading a little sharpness for robustness.

A broadcasting bug as a teaching moment. Converting the row-normalization step into an efficient, vectorized operation exposes a real and easy-to-miss PyTorch pitfall: summing across a dimension with keepdim=False collapses a column-shaped sum into a plain vector, which broadcasting rules then silently reinterpret as a row vector when it's used in a division — replicating it in the wrong direction and normalizing columns instead of rows. The code runs without any error either way; only checking that each row of the result actually sums to one reveals which version is correct. He treats this less as an aside than as a warning: broadcasting semantics deserve real study, because they fail silently rather than loudly.

Reframing bigram modeling as a neural network. Rather than storing probabilities in a table, he encodes each input character as a one-hot vector and feeds it through a single linear layer of 27 neurons with no bias and no nonlinearity, producing 27 raw outputs per input that he explicitly interprets as log-counts ("logits"). Exponentiating these gives count-like positive numbers, and normalizing them (the two steps together forming a softmax) yields the same shape of probability distribution as the counting approach — and because a one-hot vector times a weight matrix simply extracts one row of that matrix, the trained weights end up being nothing more than the log of the original count table, arrived at from a different direction. Training proceeds exactly as with micrograd: a forward pass computes probabilities and the negative-log-likelihood loss, loss.backward() populates gradients via PyTorch's autograd, and a simple update step nudges every weight against its gradient — with gradient descent alone converging to the same ~2.45 loss the explicit counting approach found directly, confirming the two are mathematically the same model reached by different roads.

Why the neural framework is worth the extra machinery. Counting works perfectly well for a single character of context because a 27x27 table is small and every cell gets plenty of data, but scaling to two, three, or ten characters of context blows the table up exponentially and starves most entries of data — a wall that counting fundamentally can't get past. The gradient-based approach doesn't have this ceiling: nothing about the loss function, the softmax, or the optimization changes as the input representation grows more complex, which is exactly what lets the same machinery scale, in later lectures, from a single input character all the way to full Transformer architectures. A closing observation ties the two approaches together further: penalizing large weights in the loss (an L2 regularization term) pushes weights toward zero, which flattens the softmax output toward uniform — mathematically the same effect as adding fake counts in the table-based version, just implemented as a tunable loss term instead of a table edit.

Starting from a completely blank notebook, this lecture builds micrograd — a roughly 150-line scalar-valued automatic differentiation engine — from first principles, then uses it to define and train a tiny multi-layer perceptron (three inputs, two hidden layers of four neurons, one output; 41 total parameters) on a four-example toy classification problem until its loss reaches essentially zero and its predictions match the targets almost exactly. The destination isn't the model — it's a complete, ground-up understanding of what every modern deep learning framework (PyTorch included) is actually doing underneath its tensors and GPUs, since he demonstrates throughout that the only difference between micrograd and production frameworks is efficiency: PyTorch does the identical math, just packaged into parallel n-dimensional arrays instead of individual scalars.

What a derivative actually measures. Rather than symbolically differentiating anything, the lecture starts from the raw definition — the limit as h approaches zero of (f(x+h) − f(x))/h — and estimates it numerically on a simple parabola, building the intuition that a derivative just measures how sensitively an output responds to a small nudge in an input, and in which direction. This same numerical-nudge approach is reused throughout as a way to sanity-check every gradient the engine computes analytically, treating agreement between the numerical estimate and the backpropagated value as the truth test for correctness.

Building the Value object and its expression graph. A Value class wraps a single scalar and overloads Python's arithmetic operators (__add__, __mul__, later power, negation, and division derived from those) so that ordinary-looking Python expressions silently construct a full computational graph: every Value remembers which operation produced it and which other Values (its children) fed into that operation. Visualizing this graph directly makes it concrete — feeding two inputs through a chain of additions and multiplications produces a small DAG whose leaves are the original inputs and whose root is the final output, with every intermediate result labeled and inspectable.

Manual backpropagation through a full expression. Working outward from a final output whose gradient with respect to itself is trivially 1, he hand-derives the gradient of every node in a small expression graph by applying the chain rule one node at a time: for any node, the gradient of the final output with respect to that node's inputs equals the node's known "local" derivative (how its output changes with respect to each input, in isolation) multiplied by the gradient already flowing into it from downstream. Working through an addition node and a multiplication node repeatedly, a clean pattern emerges: addition nodes have a local derivative of 1, so they simply copy their incoming gradient unchanged to every child (they "route" gradient); multiplication nodes have a local derivative equal to the other input's value, so they scale the incoming gradient by that value before passing it down. Every manually-derived gradient is cross-checked against a numerical nudge, and they agree.

Backpropagating through a neuron. The same machinery is applied to a biologically-inspired model of a single neuron — a weighted sum of inputs plus a bias, squashed through a tanh nonlinearity — introduced first as one atomic operation (since its forward value and derivative, 1 − tanh(x)², are both known from calculus) and later reimplemented from raw exponentiation, addition, and division primitives to prove the two approaches give bit-for-bit identical forward outputs and gradients. This demonstrates a general principle he states explicitly: the graph doesn't care whether a node represents an atomic operation like addition or an arbitrarily complex composite function — all that matters for backpropagation is that you can compute the node's output and its local derivative, a principle mirrored directly in how PyTorch lets users register entirely custom autograd functions.

Automating the backward pass. Rather than manually propagating gradients node by node, each Value gains a private _backward closure that knows how to push its own output gradient into its children according to its specific operation, and calling .backward() on the final node performs a topological sort of the graph — ensuring no node's gradient is finalized before every node that depends on it has already propagated its contribution — then walks the sorted list in reverse, invoking every node's _backward in the correct order. Along the way, a subtle and consequential bug surfaces: when a value is reused multiple times in an expression (as simple as b = a + a), naively setting a child's gradient during each backward call overwrites earlier contributions instead of combining them, giving a wrong answer (a gradient of 1 instead of the correct 2). The fix, motivated by the multivariable chain rule's requirement to sum contributions across every path from a variable to the output, is to accumulate gradients with += rather than assign them with = — a one-character difference that determines correctness anywhere a value or a set of shared weights gets used more than once, which describes essentially every real neural network.

Confirming the engine against PyTorch. The same small expression, rebuilt using real PyTorch tensors (wrapped as single-element double-precision tensors with requires_grad=True), produces identical forward values and identical gradients once .backward() is called — establishing that micrograd, despite operating on individual Python scalars instead of parallel tensor arrays, is mathematically doing exactly what a production framework does, just without the efficiency of batched, hardware-accelerated array operations.

From engine to neural network library. On top of the roughly 100-line autograd engine, an entire neural network library is built in about 50 more lines: a Neuron computes a weighted sum of its inputs plus a bias and passes it through tanh; a Layer is simply a list of independently-evaluated neurons; and a Multi-Layer Perceptron (MLP) chains layers sequentially, mirroring PyTorch's own nn.Module API closely enough that a shared parameters() method can gather every weight and bias across the whole network for updating. A four-example toy dataset (each example a 3-dimensional input paired with a target of +1 or −1) is used to define a mean-squared-error loss — a single number summarizing how far every prediction is from its target, engineered so that lower is always better and zero means perfect predictions.

Training via gradient descent, bugs and all. Calling .backward() on the loss back-propagates through all four forward passes simultaneously, populating a .grad on every one of the network's 41 parameters; nudging each parameter a small step in the negative direction of its gradient (since the gradient points toward increasing loss, and the goal is to decrease it) measurably lowers the loss on the next forward pass, and repeating this forward-backward-update cycle is gradient descent itself. Choosing a learning rate is shown to be delicate first-hand: too small converges too slowly, too large risks destabilizing or exploding the loss, though in one case the model overshoots and by luck rapidly lands in a very good spot anyway. Left deliberately uncut is a real, on-camera bug: because each .backward() call accumulates gradients onto whatever a parameter's .grad already holds rather than resetting it, omitting an explicit "zero the gradients" step before each backward pass caused gradients to compound step over step, artificially inflating the effective step size — the network still converged, but only because the toy problem was trivially easy; with anything harder, the same one-line omission (a mistake he notes he's made many times, on camera, over a career) would have derailed training entirely. The closing framing ties it all together: a neural network, no matter how large, is just a mathematical expression connecting data and parameters to a loss through additions, multiplications, and nonlinearities; gradient descent is simply following that expression's local slope downhill; and the same principles — differing only in scale, nonlinearity choice, and loss function — extend unchanged from a 41-parameter toy classifier all the way to language models with hundreds of billions of parameters.