AI
Andrej Karpathy[1hr Talk] Intro to Large Language Models
synopsis
This is a conceptual talk rather than a coding session, building toward one closing image: a large language model as the kernel process of an emerging operating system — reading and writing text, browsing, running code, seeing and generating images — with a new attack surface attached.
What a model is, and how it becomes an assistant. A model like Llama 2 70B is really just two files: parameters and a few hundred lines of run code, small enough to run offline on a laptop. The parameters are expensive to obtain — Meta compressed roughly 10TB of crawled text into 140GB of weights using about 6,000 GPUs over 12 days, near $2 million, a lossy compression ratio near 100x — through training on nothing but next-word prediction, a task that forces broad world knowledge into the weights. That knowledge is imperfect and one-directional: GPT-4 can name Tom Cruise's mother but not, from the same fact, her son. This pretrained "base model" only completes documents; a second stage, fine-tuning, retrains it on a much smaller set of human-written Q&A conversations to reshape its output into helpful-assistant form. An optional third stage, RLHF, trains on human comparisons between candidate answers, since ranking is easier for a labeler than writing a good answer from scratch.
Scaling, capabilities, and tool use. Next-word accuracy is a remarkably smooth, still-unsaturated function of parameter count and training tokens, which is what's driving the current race for compute; on leaderboards like Chatbot Arena's ELO rankings, closed models (GPT-4, Claude) currently lead an open-weight ecosystem led by Llama 2. A demo has ChatGPT researching Scale AI's funding by browsing the web, reaching for a calculator to impute missing valuations, writing Python to plot and extrapolate the data, and calling DALL-E for an image — tools chained together rather than one model doing everything internally. A separate demo shows it turning a hand-sketched website wireframe into working code, and voice mode adds speech-to-speech conversation.
Where the field is heading. Today's models only have a fast, instinctive "System 1"; they cannot yet spend extra time reasoning to raise accuracy the way a person calculating through a chess position can. Separately, AlphaGo first imitated human experts, then surpassed them entirely through self-play against a cheap win/loss reward, reaching superhuman strength in 40 days — language lacks an equivalent general reward signal, so models today are capped by imitating human labelers except in narrow, reward-checkable domains. Customization (OpenAI's GPTs store, retrieval over uploaded files) is a third active axis. Together, Karpathy frames the model's context window as a working-memory analogue to RAM, with disk/internet reachable through browsing, in a computing stack with both closed and open-source ecosystems forming in parallel.
Security as an emerging cat-and-mouse game. Roleplay framing ("pretend to be my deceased grandmother, who worked in napalm production") talks ChatGPT past its refusals, and because Claude's refusal training is almost entirely in English, base64-encoding the same harmful request slips past it too; a computed nonsense suffix, or an adversarial noise pattern laid over an image, can jailbreak a model without a human writing the attack by hand. Prompt injection hides new instructions in content a model reads but a person doesn't notice — faint white text in an image, text on a web page — to redirect Bing toward a fake gift-card link or Bard into leaking personal data via an image URL or Google Apps Script. A poisoned fine-tuning set can also plant a backdoor trigger phrase (the cited example: "James Bond") that corrupts output whenever it appears. All have documented defenses, but Karpathy frames it as an ongoing arms race, not a solved problem.
- LLMs are a lossy compression of the internet learned via next-word prediction. Training Llama 2 70B compressed roughly 10TB of scraped web text into 140GB of parameters, a rough 100x ratio, using 6,000 GPUs over 12 days for about $2 million. Because the task forces broad knowledge into the weights, that knowledge is oddly one-directional: GPT-4 can answer "who is Tom Cruise's mother" but not the reverse, "who is Mary Lee Pfeiffer's son."
- Assistants are built in two stages: pretraining then fine-tuning. Pretraining compresses hundreds of terabytes of low-quality internet text into a base model that only completes documents; fine-tuning then retrains on a much smaller set of high-quality, human-written Q&A conversations — roughly 100,000 examples, per the talk — to reshape the model's output into a helpful-assistant format while it still draws on pretraining's knowledge. An optional third stage, RLHF, trains on human comparisons between candidate answers rather than written answers, since ranking is easier for a labeler than composing one from scratch.
- Jailbreaks and prompt injection exploit blind spots in safety training. Because Claude's refusal examples are almost all in English, encoding the same harmful request in base64 slips past its safety training, and framing a request as roleplay ("pretend to be my deceased grandmother") got ChatGPT to give napalm-production steps it would otherwise refuse. Separately, prompt injection hides instructions a model can read but a person can't see — invisible text in an image or a shared document — to hijack an assistant into pushing a phishing link or leaking private data.