AI

Rather than adding a new architecture, this lecture strips one away: he takes the two-layer MLP-plus-BatchNorm character model from the previous lecture and rips out loss.backward(), replacing PyTorch's autograd with a fully hand-derived backward pass, tensor by tensor, until the same model trains to the same loss using nothing but manually computed gradients. The point isn't the result — it's that backpropagation is what he calls a "leaky abstraction," and skipping straight to autograd hides exactly the kind of bugs (a real-world example: a codebase that clips the loss instead of the gradient, silently zeroing outliers' influence) that only someone who's derived it by hand will catch.

Historical framing. He opens by noting that hand-written backward passes were standard practice barely a decade ago — showing his own 2010 Matlab code for training restricted Boltzmann machines and 2014 NumPy code for an image-sentence embedding model, both with manually implemented gradients checked against numerical estimates — before autograd engines like PyTorch's made it unnecessary, and argues that understanding the mechanism is still what separates someone who can debug a broken network from someone who can only rerun it.

Exercise 1: full atomic backpropagation. Working through the forward pass broken into its smallest pieces — log-probabilities, counts, exponentials, matrix multiplies, tanh, and every BatchNorm sub-step — he derives each gradient by hand and checks it against PyTorch's own .grad values. Two recurring principles do most of the work: local derivatives compose via the chain rule exactly as in his earlier micrograd project, and there is a duality between forward-pass operations and backward-pass shapes — summing in the forward pass (as when a bias vector or BatchNorm parameter gets broadcast, i.e., replicated, across many rows) means broadcasting the gradient back to that same shape in reverse, and vice versa; a variable reused multiple times (like a shared embedding row) must have its gradients from every use added together. Matrix multiplication derivatives, rather than being memorized, are derived by writing out a tiny 2×2 example element-by-element and pattern-matching the resulting shapes, arriving at the standard dA = dOut · Bᵀ, dB = Aᵀ · dOut results without invoking matrix calculus formalism.

Exercise 2: the analytical shortcut for cross-entropy. Rather than backpropagating through the six-plus intermediate steps of softmax and negative-log-likelihood individually, he derives on paper that the gradient of the combined loss with respect to the logits collapses to something remarkably clean: the softmax probabilities, with 1 subtracted at the position of the correct label, all divided by batch size. Visualizing this gradient row by row, he offers a physical intuition — think of it as a tug-of-war where the correct answer's probability gets pulled up and every incorrect answer's probability gets pulled down, with the two forces in each row exactly balancing (summing to zero), and the strength of each pull directly proportional to how wrong the current prediction is; a perfectly confident correct prediction produces no gradient at all.

Exercise 3: the analytical shortcut for BatchNorm. He repeats the same collapsing exercise for BatchNorm, which is considerably harder because both the mean and variance are computed from, and feed back into, every element of the batch, creating multiple gradient paths that must be summed via the chain rule before they cancel and simplify into one manageable closed-form expression — verified by matching PyTorch's own backward pass to within floating-point precision. Along the way he flags what he considers a real bug: the original BatchNorm paper (and PyTorch's own implementation) use the biased variance estimator (dividing by n) during training but switch to the unbiased, Bessel-corrected version (dividing by n−1) for the running statistics used at inference — a train/test inconsistency with no clear justification, which he resolves in his own implementation by using the unbiased estimator consistently in both places.

Exercise 4: putting it together. With every gradient now derivable from a compact, roughly 20-line backward pass — rather than the sprawling atomic version from Exercise 1 — he retrains the full network end to end using only these manually computed gradients, with loss.backward() and PyTorch's autograd entirely absent, reaching essentially the same loss and generating the same quality of name-like samples as before. The exercise's real deliverable isn't a better model; it's confidence, built layer by layer, that nothing about how these networks learn is mysterious — only, in BatchNorm's case, a bit more algebraically involved than the rest.

Staying with the same character-level MLP from the previous lecture rather than moving to recurrent networks as planned, he spends the session diagnosing and fixing exactly what can go wrong inside a neural network's activations and gradients at initialization — pushing validation loss from 2.17 down to about 2.10 through nothing but better initialization and normalization — because, as he argues, these same failure modes become far more dangerous once networks get much deeper, which is the whole reason to understand them now.

The hockey-stick loss. Fresh off training, he notices the very first loss recorded is 27, wildly higher than the ~3.29 a 27-way uniform guess should cost (negative log of 1/27). The cause: default initialization produces logits with large, arbitrary magnitudes, so the softmax comes out confidently wrong for most examples rather than diffuse; the first chunk of training is then spent simply shrinking those logits back toward zero, visible as a sharp early drop in the loss curve before real learning starts. The fix is to initialize the final layer's weights (and biases) near zero, which immediately produces the expected initial loss, removes the hockey-stick shape, and — because training time is no longer wasted correcting an artificial problem — measurably improves the final loss.

Saturated tanh units and dead neurons. A histogram of the hidden layer's activations reveals most values pinned at exactly +1 or -1 — tanh fully saturated. This matters because tanh's backward-pass gradient is (1 − output²): an output near ±1 multiplies the incoming gradient by nearly zero, effectively cutting off learning for that unit on that example. Checking column-by-column for neurons where every example saturates reveals the more serious failure mode: a "dead neuron" that never receives a gradient again and is permanently stuck — a risk he notes is shared by sigmoid (also a squashing function) and ReLU (a dead ReLU, unable to recover once knocked into producing only negative pre-activations, sometimes triggered by too-high a learning rate). The fix mirrors the logit fix: shrink the weights and biases feeding into the hidden layer so pre-activations land in a reasonable range, eliminating the saturated tail and further improving loss.

Principled initialization. Rather than hand-picking scale factors by trial and error, he derives the general rule from Kaiming/He initialization: to keep a layer's output variance equal to its input's, weights should be drawn with standard deviation equal to a gain divided by the square root of fan-in. The gain compensates for how much a given nonlinearity compresses its input distribution — 1 for a purely linear layer, but roughly 5/3 for tanh, whose squashing otherwise causes activation variance to shrink toward zero as it passes through successive layers (demonstrated directly: gain too low and deeper layers' activations collapse toward zero; gain too high and they oversaturate). He's candid that this precise tuning mattered enormously before several later innovations — residual connections, various normalization layers, and better optimizers like Adam — made networks considerably more forgiving of imperfect initialization.

Batch normalization. The core insight, introduced by a 2015 Google paper, is disarmingly direct: since what's wanted is roughly Gaussian pre-activations at every layer, why not just standardize them directly — subtract the batch's mean, divide by its standard deviation — since that operation is fully differentiable? To avoid permanently forcing a fixed distribution, this is paired with a learnable gain and bias so the network can still shift or reshape the distribution as training demands. He implements it as a drop-in layer placed after linear layers and before nonlinearities, demonstrating that it makes networks dramatically more robust to the exact scale of the preceding weights — even removing fan-in scaling entirely still trains reasonably once BatchNorm is present, though the learning rate may need retuning since BatchNorm's own backward pass changes how gradient scale relates to the weights feeding into it.

The catch is a real one: normalizing over the batch means an example's hidden state is no longer a pure function of that example — it now depends on whichever other examples happened to land in the same batch, introducing "jitter" that acts as an accidental regularizer but is also, he says, a persistent source of subtle bugs, which is why later techniques (layer, instance, and group normalization) were developed to decouple examples again. A related practical wrinkle: since BatchNorm re-centers its input, any bias term in the preceding linear layer is mathematically canceled out and wastes a parameter, so linear layers immediately before a BatchNorm should disable their bias. For inference on a single example, where batch statistics don't exist, the layer instead uses a running mean and variance accumulated via exponential moving average during training — the same approach used in PyTorch's own BatchNorm1d, whose parameters (momentum, epsilon, affine, track_running_stats) he walks through directly, noting that overly large momentum can be unstable for small batch sizes.

Diagnostics for a healthy network. Beyond the fixes, he introduces the habit of actively monitoring three things during training: histograms of forward-pass activations and backward-pass gradients across layers, watching for saturation or asymmetric shrink/growth with depth, and — most usefully — the ratio of each parameter's update size to its own data scale, plotted on a log scale over training. His rule of thumb is that this ratio should hover around 10⁻³; consistently far below suggests the learning rate is too low, far above suggests it's too high, and BatchNorm-equipped networks are shown to be considerably more forgiving of these settings than the raw linear-tanh stack was. He closes by noting these gains didn't actually beat the model's previous best loss — because the real bottleneck at this point is a context window of only three characters, not optimization — foreshadowing the shift to recurrent architectures and, eventually, Transformers.

Moving past the bigram model's fundamental limit — its lookup table grows exponentially with context length, making anything beyond a single prior character intractable — this lecture implements a multi-layer perceptron character-level language model following Bengio et al. 2003, ending with a network that takes three characters of context, embeds them, and predicts the fourth, reaching a validation loss of about 2.17 (versus 2.45 for the bigram model) and producing noticeably more name-like samples like "ham," "joes," and "lela."

The architecture. Each of 27 possible characters is mapped to a point in a low-dimensional embedding space via a lookup table C, initialized randomly and tuned through backpropagation alongside the rest of the network. He demonstrates that indexing into this table is mathematically equivalent to multiplying a one-hot vector by C — framing the embedding step as just a linear layer with no nonlinearity — before settling on the much faster direct-indexing approach. Three character embeddings are concatenated (efficiently, via .view() rather than torch.cat, since reshaping a tensor's view requires no new memory) and fed through a hidden layer with a tanh nonlinearity, then a final linear layer producing 27 logits. The motivating idea from the paper: since embeddings place similar characters or words near each other in this learned space, the model can generalize to combinations it never saw in training — if it learns "a" and "the" behave similarly, it can transfer what it learned about "the dog was running" to "a dog was running" even without that exact phrase in the data.

Practical training mechanics. He switches the hand-rolled softmax-and-negative-log-likelihood computation to PyTorch's F.cross_entropy, noting it's not just more convenient but meaningfully better: it fuses operations into a faster forward and a mathematically simpler backward pass, and critically avoids numerical overflow by internally subtracting the batch's maximum logit before exponentiating — since softmax output is invariant to a constant offset, this keeps very large logits from producing inf. Training then moves to mini-batches of 32 random examples rather than the full 228,000, trading a noisier, approximate gradient for dramatically more update steps in the same time — which turns out to converge faster overall since gradient direction, even if imperfect, is still useful.

Finding a learning rate. Rather than guessing, he sweeps candidate learning rates exponentially from 0.001 to 1 across a short run and plots the resulting loss against the learning rate's exponent, producing a characteristic valley: too low and the loss barely moves, too high and training becomes unstable. The middle of that valley — around 0.1 — gives a defensible starting point, later followed by a manual 10x decay once the loss plateaus, a two-stage schedule he's candid is "janky" compared to production practice but effective enough here.

Diagnosing underfitting versus overfitting. Splitting the data 80/10/10 into train, dev (validation), and test sets, he shows that when training and dev loss track closely (~2.3 each) but remain fairly high, the model is underfitting — too small to be memorizing, let alone generalizing well — and needs more capacity rather than regularization. Tripling the hidden layer's neuron count barely moved the needle, but increasing the character embedding dimensionality from 2 to 10 (with the hidden layer trimmed back to 200 units) dropped the loss meaningfully further, revealing that the 2-dimensional embedding space, not the hidden layer, was the actual bottleneck restricting how much distinct information the network could represent per character. Visualizing the trained 2D embeddings before scaling them up makes this concrete: vowels cluster tightly together, ordinary consonants occupy a shared middle region, and outliers like "q" and the special "." token sit far off on their own — visible evidence that the space is encoding real structure rather than sitting at its random initialization. The chapter closes by explicitly flagging the knobs left untuned — embedding size, hidden layer width, context length, batch size, and learning rate schedule — as an open invitation to keep improving past the 2.17 validation loss reached here.

This lecture builds the first and simplest member of the makemore family — a bigram, character-level name generator — twice over: once by directly counting character-pair frequencies, and once by training a single-layer neural network with gradient descent, arriving at the identical ~2.45 loss and identical output samples both ways. The destination isn't a strong model (bigram statistics are, in his words, "just really terrible" — producing garbled near-names rather than good ones) but a fully worked bridge between classical counting-based statistical modeling and the gradient-based neural framework that every later lecture in the series builds on.

Counting bigrams. Starting from a 32,000-name dataset, he treats each name as a sequence of bigrams — consecutive character pairs — wrapping every word with a special start/end token (later unified into a single "." token) so the model also learns which characters are likely to begin or end a name. Every one of the roughly 27x27 possible pairs (26 letters plus the special token) gets a count, tallied into a PyTorch tensor and visualized as a heatmap, revealing structure like "N" being an extremely common ending character.

From counts to a generative model. Normalizing each row of the count matrix into a probability distribution turns raw statistics into something samplable: starting from the special start token, torch.multinomial draws a next character according to that row's probabilities, then repeats using the new character's row, until the end token is drawn. This produces recognizably name-adjacent gibberish ("Keanu," "O'Reilly"-ish fragments), which he confirms is genuinely better than a uniform baseline, even though it's still clearly a weak model since it only ever looks one character back.

Measuring model quality and its sharp edges. To turn "how good is this model" into a single number, he introduces the likelihood (the product of the probabilities the model assigns to every actual next character in the training data) and its practical cousins: log-likelihood (summing logs instead of multiplying tiny numbers) and negative log-likelihood, averaged, which becomes the loss to minimize — lower is better, zero is perfect. A sharp edge appears immediately: querying the model on a plausible but unseen bigram (a name ending in "...jq") returns exactly zero probability, and log(0) is negative infinity, breaking the loss entirely. The fix, "model smoothing," adds a small fake count to every bigram before normalizing, guaranteeing no probability is ever exactly zero — trading a little sharpness for robustness.

A broadcasting bug as a teaching moment. Converting the row-normalization step into an efficient, vectorized operation exposes a real and easy-to-miss PyTorch pitfall: summing across a dimension with keepdim=False collapses a column-shaped sum into a plain vector, which broadcasting rules then silently reinterpret as a row vector when it's used in a division — replicating it in the wrong direction and normalizing columns instead of rows. The code runs without any error either way; only checking that each row of the result actually sums to one reveals which version is correct. He treats this less as an aside than as a warning: broadcasting semantics deserve real study, because they fail silently rather than loudly.

Reframing bigram modeling as a neural network. Rather than storing probabilities in a table, he encodes each input character as a one-hot vector and feeds it through a single linear layer of 27 neurons with no bias and no nonlinearity, producing 27 raw outputs per input that he explicitly interprets as log-counts ("logits"). Exponentiating these gives count-like positive numbers, and normalizing them (the two steps together forming a softmax) yields the same shape of probability distribution as the counting approach — and because a one-hot vector times a weight matrix simply extracts one row of that matrix, the trained weights end up being nothing more than the log of the original count table, arrived at from a different direction. Training proceeds exactly as with micrograd: a forward pass computes probabilities and the negative-log-likelihood loss, loss.backward() populates gradients via PyTorch's autograd, and a simple update step nudges every weight against its gradient — with gradient descent alone converging to the same ~2.45 loss the explicit counting approach found directly, confirming the two are mathematically the same model reached by different roads.

Why the neural framework is worth the extra machinery. Counting works perfectly well for a single character of context because a 27x27 table is small and every cell gets plenty of data, but scaling to two, three, or ten characters of context blows the table up exponentially and starves most entries of data — a wall that counting fundamentally can't get past. The gradient-based approach doesn't have this ceiling: nothing about the loss function, the softmax, or the optimization changes as the input representation grows more complex, which is exactly what lets the same machinery scale, in later lectures, from a single input character all the way to full Transformer architectures. A closing observation ties the two approaches together further: penalizing large weights in the loss (an L2 regularization term) pushes weights toward zero, which flattens the softmax output toward uniform — mathematically the same effect as adding fake counts in the table-based version, just implemented as a tunable loss term instead of a table edit.

Starting from a completely blank notebook, this lecture builds micrograd — a roughly 150-line scalar-valued automatic differentiation engine — from first principles, then uses it to define and train a tiny multi-layer perceptron (three inputs, two hidden layers of four neurons, one output; 41 total parameters) on a four-example toy classification problem until its loss reaches essentially zero and its predictions match the targets almost exactly. The destination isn't the model — it's a complete, ground-up understanding of what every modern deep learning framework (PyTorch included) is actually doing underneath its tensors and GPUs, since he demonstrates throughout that the only difference between micrograd and production frameworks is efficiency: PyTorch does the identical math, just packaged into parallel n-dimensional arrays instead of individual scalars.

What a derivative actually measures. Rather than symbolically differentiating anything, the lecture starts from the raw definition — the limit as h approaches zero of (f(x+h) − f(x))/h — and estimates it numerically on a simple parabola, building the intuition that a derivative just measures how sensitively an output responds to a small nudge in an input, and in which direction. This same numerical-nudge approach is reused throughout as a way to sanity-check every gradient the engine computes analytically, treating agreement between the numerical estimate and the backpropagated value as the truth test for correctness.

Building the Value object and its expression graph. A Value class wraps a single scalar and overloads Python's arithmetic operators (__add__, __mul__, later power, negation, and division derived from those) so that ordinary-looking Python expressions silently construct a full computational graph: every Value remembers which operation produced it and which other Values (its children) fed into that operation. Visualizing this graph directly makes it concrete — feeding two inputs through a chain of additions and multiplications produces a small DAG whose leaves are the original inputs and whose root is the final output, with every intermediate result labeled and inspectable.

Manual backpropagation through a full expression. Working outward from a final output whose gradient with respect to itself is trivially 1, he hand-derives the gradient of every node in a small expression graph by applying the chain rule one node at a time: for any node, the gradient of the final output with respect to that node's inputs equals the node's known "local" derivative (how its output changes with respect to each input, in isolation) multiplied by the gradient already flowing into it from downstream. Working through an addition node and a multiplication node repeatedly, a clean pattern emerges: addition nodes have a local derivative of 1, so they simply copy their incoming gradient unchanged to every child (they "route" gradient); multiplication nodes have a local derivative equal to the other input's value, so they scale the incoming gradient by that value before passing it down. Every manually-derived gradient is cross-checked against a numerical nudge, and they agree.

Backpropagating through a neuron. The same machinery is applied to a biologically-inspired model of a single neuron — a weighted sum of inputs plus a bias, squashed through a tanh nonlinearity — introduced first as one atomic operation (since its forward value and derivative, 1 − tanh(x)², are both known from calculus) and later reimplemented from raw exponentiation, addition, and division primitives to prove the two approaches give bit-for-bit identical forward outputs and gradients. This demonstrates a general principle he states explicitly: the graph doesn't care whether a node represents an atomic operation like addition or an arbitrarily complex composite function — all that matters for backpropagation is that you can compute the node's output and its local derivative, a principle mirrored directly in how PyTorch lets users register entirely custom autograd functions.

Automating the backward pass. Rather than manually propagating gradients node by node, each Value gains a private _backward closure that knows how to push its own output gradient into its children according to its specific operation, and calling .backward() on the final node performs a topological sort of the graph — ensuring no node's gradient is finalized before every node that depends on it has already propagated its contribution — then walks the sorted list in reverse, invoking every node's _backward in the correct order. Along the way, a subtle and consequential bug surfaces: when a value is reused multiple times in an expression (as simple as b = a + a), naively setting a child's gradient during each backward call overwrites earlier contributions instead of combining them, giving a wrong answer (a gradient of 1 instead of the correct 2). The fix, motivated by the multivariable chain rule's requirement to sum contributions across every path from a variable to the output, is to accumulate gradients with += rather than assign them with = — a one-character difference that determines correctness anywhere a value or a set of shared weights gets used more than once, which describes essentially every real neural network.

Confirming the engine against PyTorch. The same small expression, rebuilt using real PyTorch tensors (wrapped as single-element double-precision tensors with requires_grad=True), produces identical forward values and identical gradients once .backward() is called — establishing that micrograd, despite operating on individual Python scalars instead of parallel tensor arrays, is mathematically doing exactly what a production framework does, just without the efficiency of batched, hardware-accelerated array operations.

From engine to neural network library. On top of the roughly 100-line autograd engine, an entire neural network library is built in about 50 more lines: a Neuron computes a weighted sum of its inputs plus a bias and passes it through tanh; a Layer is simply a list of independently-evaluated neurons; and a Multi-Layer Perceptron (MLP) chains layers sequentially, mirroring PyTorch's own nn.Module API closely enough that a shared parameters() method can gather every weight and bias across the whole network for updating. A four-example toy dataset (each example a 3-dimensional input paired with a target of +1 or −1) is used to define a mean-squared-error loss — a single number summarizing how far every prediction is from its target, engineered so that lower is always better and zero means perfect predictions.

Training via gradient descent, bugs and all. Calling .backward() on the loss back-propagates through all four forward passes simultaneously, populating a .grad on every one of the network's 41 parameters; nudging each parameter a small step in the negative direction of its gradient (since the gradient points toward increasing loss, and the goal is to decrease it) measurably lowers the loss on the next forward pass, and repeating this forward-backward-update cycle is gradient descent itself. Choosing a learning rate is shown to be delicate first-hand: too small converges too slowly, too large risks destabilizing or exploding the loss, though in one case the model overshoots and by luck rapidly lands in a very good spot anyway. Left deliberately uncut is a real, on-camera bug: because each .backward() call accumulates gradients onto whatever a parameter's .grad already holds rather than resetting it, omitting an explicit "zero the gradients" step before each backward pass caused gradients to compound step over step, artificially inflating the effective step size — the network still converged, but only because the toy problem was trivially easy; with anything harder, the same one-line omission (a mistake he notes he's made many times, on camera, over a career) would have derailed training entirely. The closing framing ties it all together: a neural network, no matter how large, is just a mathematical expression connecting data and parameters to a loss through additions, multiplications, and nonlinearities; gradient descent is simply following that expression's local slope downhill; and the same principles — differing only in scale, nonlinearity choice, and loss function — extend unchanged from a 41-parameter toy classifier all the way to language models with hundreds of billions of parameters.