Starting from a completely blank notebook, this lecture builds micrograd — a roughly 150-line scalar-valued automatic differentiation engine — from first principles, then uses it to define and train a tiny multi-layer perceptron (three inputs, two hidden layers of four neurons, one output; 41 total parameters) on a four-example toy classification problem until its loss reaches essentially zero and its predictions match the targets almost exactly. The destination isn't the model — it's a complete, ground-up understanding of what every modern deep learning framework (PyTorch included) is actually doing underneath its tensors and GPUs, since he demonstrates throughout that the only difference between micrograd and production frameworks is efficiency: PyTorch does the identical math, just packaged into parallel n-dimensional arrays instead of individual scalars.
What a derivative actually measures. Rather than symbolically differentiating anything, the lecture starts from the raw definition — the limit as h approaches zero of (f(x+h) − f(x))/h — and estimates it numerically on a simple parabola, building the intuition that a derivative just measures how sensitively an output responds to a small nudge in an input, and in which direction. This same numerical-nudge approach is reused throughout as a way to sanity-check every gradient the engine computes analytically, treating agreement between the numerical estimate and the backpropagated value as the truth test for correctness.
Building the Value object and its expression graph. A Value class wraps a single scalar and overloads Python's arithmetic operators (__add__, __mul__, later power, negation, and division derived from those) so that ordinary-looking Python expressions silently construct a full computational graph: every Value remembers which operation produced it and which other Values (its children) fed into that operation. Visualizing this graph directly makes it concrete — feeding two inputs through a chain of additions and multiplications produces a small DAG whose leaves are the original inputs and whose root is the final output, with every intermediate result labeled and inspectable.
Manual backpropagation through a full expression. Working outward from a final output whose gradient with respect to itself is trivially 1, he hand-derives the gradient of every node in a small expression graph by applying the chain rule one node at a time: for any node, the gradient of the final output with respect to that node's inputs equals the node's known "local" derivative (how its output changes with respect to each input, in isolation) multiplied by the gradient already flowing into it from downstream. Working through an addition node and a multiplication node repeatedly, a clean pattern emerges: addition nodes have a local derivative of 1, so they simply copy their incoming gradient unchanged to every child (they "route" gradient); multiplication nodes have a local derivative equal to the other input's value, so they scale the incoming gradient by that value before passing it down. Every manually-derived gradient is cross-checked against a numerical nudge, and they agree.
Backpropagating through a neuron. The same machinery is applied to a biologically-inspired model of a single neuron — a weighted sum of inputs plus a bias, squashed through a tanh nonlinearity — introduced first as one atomic operation (since its forward value and derivative, 1 − tanh(x)², are both known from calculus) and later reimplemented from raw exponentiation, addition, and division primitives to prove the two approaches give bit-for-bit identical forward outputs and gradients. This demonstrates a general principle he states explicitly: the graph doesn't care whether a node represents an atomic operation like addition or an arbitrarily complex composite function — all that matters for backpropagation is that you can compute the node's output and its local derivative, a principle mirrored directly in how PyTorch lets users register entirely custom autograd functions.
Automating the backward pass. Rather than manually propagating gradients node by node, each Value gains a private _backward closure that knows how to push its own output gradient into its children according to its specific operation, and calling .backward() on the final node performs a topological sort of the graph — ensuring no node's gradient is finalized before every node that depends on it has already propagated its contribution — then walks the sorted list in reverse, invoking every node's _backward in the correct order. Along the way, a subtle and consequential bug surfaces: when a value is reused multiple times in an expression (as simple as b = a + a), naively setting a child's gradient during each backward call overwrites earlier contributions instead of combining them, giving a wrong answer (a gradient of 1 instead of the correct 2). The fix, motivated by the multivariable chain rule's requirement to sum contributions across every path from a variable to the output, is to accumulate gradients with += rather than assign them with = — a one-character difference that determines correctness anywhere a value or a set of shared weights gets used more than once, which describes essentially every real neural network.
Confirming the engine against PyTorch. The same small expression, rebuilt using real PyTorch tensors (wrapped as single-element double-precision tensors with requires_grad=True), produces identical forward values and identical gradients once .backward() is called — establishing that micrograd, despite operating on individual Python scalars instead of parallel tensor arrays, is mathematically doing exactly what a production framework does, just without the efficiency of batched, hardware-accelerated array operations.
From engine to neural network library. On top of the roughly 100-line autograd engine, an entire neural network library is built in about 50 more lines: a Neuron computes a weighted sum of its inputs plus a bias and passes it through tanh; a Layer is simply a list of independently-evaluated neurons; and a Multi-Layer Perceptron (MLP) chains layers sequentially, mirroring PyTorch's own nn.Module API closely enough that a shared parameters() method can gather every weight and bias across the whole network for updating. A four-example toy dataset (each example a 3-dimensional input paired with a target of +1 or −1) is used to define a mean-squared-error loss — a single number summarizing how far every prediction is from its target, engineered so that lower is always better and zero means perfect predictions.
Training via gradient descent, bugs and all. Calling .backward() on the loss back-propagates through all four forward passes simultaneously, populating a .grad on every one of the network's 41 parameters; nudging each parameter a small step in the negative direction of its gradient (since the gradient points toward increasing loss, and the goal is to decrease it) measurably lowers the loss on the next forward pass, and repeating this forward-backward-update cycle is gradient descent itself. Choosing a learning rate is shown to be delicate first-hand: too small converges too slowly, too large risks destabilizing or exploding the loss, though in one case the model overshoots and by luck rapidly lands in a very good spot anyway. Left deliberately uncut is a real, on-camera bug: because each .backward() call accumulates gradients onto whatever a parameter's .grad already holds rather than resetting it, omitting an explicit "zero the gradients" step before each backward pass caused gradients to compound step over step, artificially inflating the effective step size — the network still converged, but only because the toy problem was trivially easy; with anything harder, the same one-line omission (a mistake he notes he's made many times, on camera, over a career) would have derailed training entirely. The closing framing ties it all together: a neural network, no matter how large, is just a mathematical expression connecting data and parameters to a loss through additions, multiplications, and nonlinearities; gradient descent is simply following that expression's local slope downhill; and the same principles — differing only in scale, nonlinearity choice, and loss function — extend unchanged from a 41-parameter toy classifier all the way to language models with hundreds of billions of parameters.