Souk's Notes - Unofficial Summaries

Unofficial, machine-generated summaries, as of 2026-11-10.

Technology and FriendsStephen Rose on New Features in Microsoft Teams [TECHNOLOGYANDFRIENDS 727]
synopsis

Rose, a senior product marketing manager for Microsoft Teams and host of Inside Microsoft Teams, walks through the previous week's Ignite announcements. He sorts them into "evolution" and "revolution", and the revolution is Teams Premium, a paid layer on top of Teams that he insists takes nothing away. He counts 450 Teams updates since Ignite 2021.

Evolution. Teams chats are coming to Outlook. Channels get pop-out and a "Channels 2.0" layout, and Schedule Send times a chat to land during the recipient's working hours.

Teams Premium. Rose says it is due next spring, beyond E3 or E5. It covers personalization (branding, custom backgrounds) and Intelligent Recap, which sends mentions and assigned action items after a meeting. Security covers forwarding and recording controls, traceable watermarks, and IT-set meeting templates per department. Live translation is the one feature moving into it.

For everyone. Excel Live joins PowerPoint Live, which Rose urges everyone to use for presenter control and roughly 20% of a screen share's bandwidth. IntelliFrame uses a 360 camera on the table to show each person in a room individually; he says a Yealink device is out and a Logitech one is coming. Work hours and location in calendars support hybrid "water cooler moments".

Avatars. Rose demonstrates avatars and their reactions. He cites school districts where under-18s may not turn cameras on, people uncomfortable on camera, and users in airports or rooms with sensitive whiteboards. He suggests outsourced help desks may adopt them.

  • Teams Premium is a paid add-on that removes nothing. Rose, a senior product marketing manager for Teams at Microsoft, says it was announced at the previous week's Ignite as a separate service beyond E3 or E5, due "next spring", covering personalization, intelligence and security. He says it sits on top of existing Teams; the one feature moving into it is live translation, which some advanced preview customers had already been using.
  • Intelligent Recap is meant to replace re-watching a missed meeting. Rose says it will send a summary after a transcribed meeting with each mention of you or your project and any action items assigned to you, so someone booked into three meetings at once can catch up on the two they missed.
  • Premium lets IT set meeting security by department. Rose describes per-meeting controls: no forwarding, named attendees only, no recording, and a watermark carrying a unique code so a photographed screen can be traced back to its meeting. Meeting templates let IT make, for example, HR meetings restrictive by default while sales meetings stay open.
  • Share decks with PowerPoint Live rather than screen sharing. Rose says the presenter keeps control of the deck, sees speaker notes and attendees, and viewers can click links or pop the content out. He puts the bandwidth at about 20% of a full screen share because only changed pixels redraw. Excel Live brings the same to spreadsheets, where each viewer can zoom or filter without changing anyone else's view.
  • Work hours and location in calendars restore chance office encounters. Rose says marking which days you are in the office lets colleagues book in-person time on those days, recovering the "water cooler moments" lost to remote work. He goes in on Thursdays and will add a day if people he wants to see are in, so that a commute is not wasted on an empty office.
Scott HanselmanTargeting Linux with .NET 6/7 and WSL (Windows Subsystem for Linux) on Windows 11
synopsis

Hanselman, speaking as part of the Microsoft team behind WSL ("we own the kernel here"), argues that a Windows 11 machine is a comfortable place to build .NET apps for Linux: Linux runs side by side with Windows, the tools cross between them without friction, and the result ships to Linux containers in the cloud.

Setup. He starts by making Windows Terminal the default terminal, then runs wsl --install, which replaces the old manual feature toggling. WSL uses a lightweight utility VM rather than Hyper-V, so it runs on Windows Home. Distros download from the Windows Store as tar files; he adds Oracle Linux 8.5 in seconds alongside Ubuntu, Kali and Debian, and wsl --list -v shows which are running or stopped.

Integration. Windows drives mount under /mnt/c, though he warns that working there is slower than on the Linux file system. In the other direction, explorer.exe . opens the Linux home folder through \\wsl.localhost, Notepad edits Linux files with line endings intact, and Linux GUI apps such as GIMP run in their own windows.

.NET on Linux. He recommends installing .NET from the official package archives, Ubuntu's or Microsoft's, which Microsoft supports. A dotnet new console app builds into a 142K Linux executable that needs only the runtime. .NET 6 is current, and .NET 7 is "a few weeks" away.

VS Code and the network. code . splits VS Code into a client and a server: the editor runs on Windows while the C# language server and debugger run in WSL. An ASP.NET app on Linux localhost port 7036 opens in the Windows browser because a WSL host process forwards the port from the internal 172.21 network.

Containers and deployment. Docker Desktop runs on WSL, so images are shared between the two sides. He builds his Hanselminutes site as a container and shows it running on Linux in Azure, deployed through Azure DevOps or GitHub Actions.

  • Getting Linux onto Windows is now one command. Hanselman shows wsl --install doing the Windows-features toggling and reboots that used to be manual, running a lightweight utility VM rather than Hyper-V so it works on Windows Home, which he says was deliberate so students without Pro can use it. Distros come from the Windows Store as tar files, as many as you like (he has Ubuntu 20.04, Kali, Debian and installs Oracle Linux 8.5 live), and he recommends the Store's WSL preview for updates to wsl.exe.
  • WSL is integrated with Windows, not a VM-in-a-box. Windows drives appear under /mnt/c, and Linux distros appear in Explorer via \\wsl.localhost; notepad.exe and explorer.exe . launch from the Linux shell, line endings survive editing on either side, and Linux GUI apps such as GIMP run in their own windows. His caveat: work on /mnt/c is slower than on the native Linux file system because it is "effectively" a network share, so keep code under /home.
  • .NET on Linux is just another distro package. He recommends the official package archives (Ubuntu's own, or packages.microsoft.com, which Microsoft supports for several distros) over tarballs or scripts, and builds a console app that is a 142K Linux executable needing only the runtime. The demo uses .NET 6, with .NET 7 described as "coming out in a few weeks" and .NET Conf on November 10.
  • The dev loop runs in Linux but feels local. code . installs a VS Code server in WSL, so the C# language server and debugger run on Linux while the editor is the Windows app. An ASP.NET app listening on Linux localhost (port 7036, on an internal 172.21 network) opens in the Windows browser because a WSL host process forwards the port automatically, which he calls the principle of least surprise.
  • Build on Windows, deploy to Linux containers in the cloud. Docker Desktop uses WSL, so the same images are visible from Windows and Linux; he builds his Hanselminutes site (862 episodes at the time) as a container and shows it running on Linux in Azure, deployed by an Azure DevOps pipeline that ran in 1 minute 36 seconds, with GitHub Actions as the alternative.
Crosstalk SolutionsThe best 3D print ever… #unifi #ubiquiti #ufo
  • The captions hold only song lyrics, no spoken content. The 27-second short's caption track is a few lines of a song ("There's a starman waiting in the sky…"); nothing about the 3D print or UniFi gear named in the title is said.
Andrej KarpathyBuilding makemore Part 4: Becoming a Backprop Ninja
synopsis

Rather than adding a new architecture, this lecture strips one away: he takes the two-layer MLP-plus-BatchNorm character model from the previous lecture and rips out loss.backward(), replacing PyTorch's autograd with a fully hand-derived backward pass, tensor by tensor, until the same model trains to the same loss using nothing but manually computed gradients. The point isn't the result — it's that backpropagation is what he calls a "leaky abstraction," and skipping straight to autograd hides exactly the kind of bugs (a real-world example: a codebase that clips the loss instead of the gradient, silently zeroing outliers' influence) that only someone who's derived it by hand will catch.

Historical framing. He opens by noting that hand-written backward passes were standard practice barely a decade ago — showing his own 2010 Matlab code for training restricted Boltzmann machines and 2014 NumPy code for an image-sentence embedding model, both with manually implemented gradients checked against numerical estimates — before autograd engines like PyTorch's made it unnecessary, and argues that understanding the mechanism is still what separates someone who can debug a broken network from someone who can only rerun it.

Exercise 1: full atomic backpropagation. Working through the forward pass broken into its smallest pieces — log-probabilities, counts, exponentials, matrix multiplies, tanh, and every BatchNorm sub-step — he derives each gradient by hand and checks it against PyTorch's own .grad values. Two recurring principles do most of the work: local derivatives compose via the chain rule exactly as in his earlier micrograd project, and there is a duality between forward-pass operations and backward-pass shapes — summing in the forward pass (as when a bias vector or BatchNorm parameter gets broadcast, i.e., replicated, across many rows) means broadcasting the gradient back to that same shape in reverse, and vice versa; a variable reused multiple times (like a shared embedding row) must have its gradients from every use added together. Matrix multiplication derivatives, rather than being memorized, are derived by writing out a tiny 2×2 example element-by-element and pattern-matching the resulting shapes, arriving at the standard dA = dOut · Bᵀ, dB = Aᵀ · dOut results without invoking matrix calculus formalism.

Exercise 2: the analytical shortcut for cross-entropy. Rather than backpropagating through the six-plus intermediate steps of softmax and negative-log-likelihood individually, he derives on paper that the gradient of the combined loss with respect to the logits collapses to something remarkably clean: the softmax probabilities, with 1 subtracted at the position of the correct label, all divided by batch size. Visualizing this gradient row by row, he offers a physical intuition — think of it as a tug-of-war where the correct answer's probability gets pulled up and every incorrect answer's probability gets pulled down, with the two forces in each row exactly balancing (summing to zero), and the strength of each pull directly proportional to how wrong the current prediction is; a perfectly confident correct prediction produces no gradient at all.

Exercise 3: the analytical shortcut for BatchNorm. He repeats the same collapsing exercise for BatchNorm, which is considerably harder because both the mean and variance are computed from, and feed back into, every element of the batch, creating multiple gradient paths that must be summed via the chain rule before they cancel and simplify into one manageable closed-form expression — verified by matching PyTorch's own backward pass to within floating-point precision. Along the way he flags what he considers a real bug: the original BatchNorm paper (and PyTorch's own implementation) use the biased variance estimator (dividing by n) during training but switch to the unbiased, Bessel-corrected version (dividing by n−1) for the running statistics used at inference — a train/test inconsistency with no clear justification, which he resolves in his own implementation by using the unbiased estimator consistently in both places.

Exercise 4: putting it together. With every gradient now derivable from a compact, roughly 20-line backward pass — rather than the sprawling atomic version from Exercise 1 — he retrains the full network end to end using only these manually computed gradients, with loss.backward() and PyTorch's autograd entirely absent, reaching essentially the same loss and generating the same quality of name-like samples as before. The exercise's real deliverable isn't a better model; it's confidence, built layer by layer, that nothing about how these networks learn is mysterious — only, in BatchNorm's case, a bit more algebraically involved than the rest.

  • Backpropagation is a "leaky abstraction," and not understanding it produces bugs autograd won't catch. He cites a real codebase that clipped the loss at a maximum value instead of clipping gradients, which silently zeroes the gradient of any outlier example rather than bounding its influence — exactly the kind of subtle error someone who only calls .backward() would never notice.
  • Whenever the forward pass sums a value, the backward pass must broadcast its gradient back across that same dimension, and vice versa. He calls this a duality: a variable that gets replicated (broadcast) in the forward pass — like a bias vector added to every row — needs its incoming gradients summed across that dimension in the backward pass, mirroring exactly how micrograd sums gradients arriving at a reused node.
  • The gradient of softmax-plus-cross-entropy loss collapses, after cancellation, to simply (probabilities minus one-hot-correct-label) divided by batch size. He frames this analytically-derived result as a physical "push and pull": each row's forces sum to zero, so probability is pulled down at incorrect positions and pushed up at the correct one by an amount exactly proportional to how wrong the prediction currently is, with no force at all once a prediction is already perfect.
  • BatchNorm's original paper and PyTorch's own implementation use inconsistent variance estimators between training and inference. Training normalizes with the biased estimator (dividing by n) while the running statistics used at test time apply the unbiased, Bessel-corrected estimator (dividing by n−1) — a train/test mismatch he considers a genuine bug and works around by using the unbiased estimator consistently in both phases of his own implementation.
Technology and FriendsChristina Aldan on Brain Performance and Memory [TECHNOLOGYANDFRIENDS 726]
  • Memory is physically trainable, like a muscle. Aldan, who describes her evidence as largely anecdotal, says the brain stores data in chunks across six memory subsystems, and that left-brain/right-brain works only as a metaphor. Repetition strengthens neural pathways through thicker myelin sheaths and more protein synthesis. She recommends practising one technique for three months before moving to the next.
  • Bilateral stimulation while holding a thought aids recall. Her "shoot the thumb" exercise moves the thumbs back and forth while you hold a memory in mind, which she says works both hemispheres and speeds transfer between them. She stresses it does nothing unless you are actively thinking of the thing.
  • The brain remembers patterns but rejects overload. Aldan, a brand strategist, says cadence, tone and repetition make things stick, as with phone numbers read in the familiar 3-3-4 grouping. She says about five or more simultaneous stimuli overwhelm people, which is why she steers clients away from busy websites.
  • Association anchors new information to what you already know. Her workshop ends with an hour building a Sherlock Holmes-style "mind palace". She also advises taking a test in the same conditions you studied in, down to the music or the lucky hoodie.
  • Interruptions are the number one reason people forget. Aldan says a developer juggling layers of work drops them all at "can I have five minutes?" and needs about an hour to recover. Her fix is to set boundaries ("just a moment, let me finish this"), including with yourself against phone and social-media distractions.
Scott HanselmanHow to setup the Nightscout Diabetes Management Open Source Server on Azure FOR FREE
  • Nightscout can run on Azure for nothing, indefinitely. Hanselman, a type 1 diabetic, pairs Cosmos DB as the database with an App Service web app, both on free tiers, filling what he says is a gap in reliable write-ups for Azure. He frames this as a first run-through ("a beta test"), and says he himself pays for his own Nightscout hosting.
  • Lock the database to its free tier before creating it. He ticks "apply free tier discount" and "limit your account", which caps it at 25 GB and 1,000 resource units, more than enough for Nightscout, and keeps the database named test at a maximum of 1,000 RU because Nightscout still expects that name. In the Node.js connection string he replaces maxIdleTimeMS with socketTimeoutMS to work around what he calls a small Nightscout issue.
  • Deploy the website from a pinned Docker image on the free plan. He creates a Linux web app from the Nightscout team's Docker Hub image nightscout/cgm-remote-monitor, recommending the specific version 14.2.6 rather than latest, on the free F1 plan. The paid tiers he shows cost about $80 and $13 a month, and he calls the $13 one a reliable, inexpensive option; the web app should sit in the same region as the database.
  • Configuration is a handful of app settings, each save restarting the site. The essentials are DB_URI with the edited connection string and API_SECRET, which failed until he made it at least 12 characters; optional settings cover plugins, enabled features and the Dexcom bridge credentials. Each save restarts Nightscout, so the site needs a moment before refreshing works.
  • Confirm it worked by checking for the seven collections. He looks for food, settings, treatments, device status, activity, profile and entries in the database, and says to restart the web app or start over if they are missing; first launch then goes through the profile editor, authenticated with the API secret.
GCastUsing the Microsoft Graph API [GCast 140]
  • Graph API is one REST interface to everything in Microsoft 365. Giard describes it as a plain REST API called over HTTP or HTTPS that can read, create, update and delete user accounts, Active Directory information, email, OneDrive files and calendar data; he calls it simple to use and well documented, pointing to the "Microsoft Graph API v1.0 endpoint reference".
  • The call is easy; the prerequisites are the real work. Every call needs a bearer token saying which account it runs under, issued by a trusted authority such as Azure Active Directory from a tenant ID, an application ID and a client secret. That account must already hold the permission the call needs (user read access to read users, create permission to create email), and getting the ID, secret and permissions means registering the app with Azure Active Directory, steps he covers in episodes 137, 138 and 139.
  • Decoding the token shows exactly which permissions it carries. In the demo, an app registration in a sample Contoso tenant was granted User.Read.All, and a token generated in Postman by an HTTP POST to a URI containing the tenant ID, passing the client ID and secret, was pasted into jwt.ms, which showed User.Read.All as its only role.
  • A GET to the users endpoint returns the tenant's users. Sending an HTTP GET to https://graph.microsoft.com/v1.0/users with the token set as a bearer token under Authorization returned HTTP 200 and the list of users. Giard says a call lacking permission or using an invalid token gets an unauthorized response ("a 401, I think"), and that a valid token with the right permissions can be reused for as many calls as you like.
Crosstalk SolutionsFinally the END of Robocalls??
  • Regulators have begun expelling non-compliant carriers from phone networks. As reported in FCC tweets and an fcc.gov notice, its Robocall Response Team issued first-of-their-kind Enforcement Bureau orders to begin removing seven named voice service providers from the Robocall Mitigation Database unless they show compliance, including STIR/SHAKEN across their IP networks. Chairwoman Jessica Rosenworcel wrote that "fines alone aren't enough."
  • Removal means every other carrier must drop that provider's traffic. The host reads it as carriers such as Verizon and Bandwidth.com getting lists of IP addresses to block outright, legitimate calls included, which he expects will effectively kill small telcos that profited from terminating robocalls. He calls it the strongest enforcement he has seen.
  • The host shut his own SIP business rather than comply. He says call-termination infrastructure, identity certificates and digitally signing every call had too steep a learning curve for the revenue, and many other small carriers left too. He still welcomes the crackdown because so many robocalls are scams; he points to his earlier STIR/SHAKEN video with Tony Lewis, CEO of ClearlyIP.
GCastCreating a JWT Bearer Token with Azure Active Directory [GCast 139]
  • A bearer token identifies the account an API request runs under. Giard describes it as an OAuth standard issued by a trusted authority; Azure Active Directory issues it as a JSON web token (JWT) carrying claims about the account, including its permissions. The account must already hold permission for whatever the API does, such as reading OneDrive documents or sending email through Microsoft Graph.
  • Three values come from the app registration. These are the client ID, the tenant ID (on the overview page and the registration) and a client secret, plus Graph application permission User.Read.All with admin consent, as in episodes 137 and 138. He creates a new secret expiring in six months and repeats that its value is shown only once and works as a password, best kept in Azure Key Vault in production.
  • The token comes from one form-encoded POST to the tenant's endpoint. Using Postman (free at postman.com), Giard posts to https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token as x-www-form-urlencoded data with four fields: client_id, client_secret, scope and grant_type. For Microsoft Graph the scope is https://graph.microsoft.com/.default and the grant type client_credentials, which he says are the same every time; the client ID and secret change.
  • The returned access token is encoded JSON that jwt.ms can decode. The request returned HTTP 200 and an access token. Decoded at jwt.ms, its aud claim names Microsoft Graph as the intended recipient and its roles claim lists User.Read.All.
  • Changing permissions does not update an already-issued token. Giard says that if you add a permission (his example is bookmark.read.all), the existing token will not carry it; you must send the POST again, from Postman or from code, to get a token that includes it.
GCastGranting Azure Active Directory API Permissions to an App Registration [GCast 138]
  • Permissions are granted on the app registration's API permissions page. This builds on the registration from episode 137. By default a registration carries the delegated permission User.Read; Giard explains delegated permissions as continuing a context the caller is already running under.
  • Application permissions make calls run as the registration's own account. Giard chooses Add a permission, then Microsoft Graph, then Application permissions, so an API call is explicitly assigned the account tied to the registration. He advises being specific because Graph covers users, email, Teams, OneDrive and more, and he picks User.Read.All under User to read all user profiles without write access.
  • User.Read.All needs admin consent before it takes effect. The portal shows a warning that admin consent has not been granted; logged on as an administrator, Giard clicks Grant admin consent and confirms, and a check mark replaces the warning. He calls the process simple, with the catch that you must be an administrator.
Technology and FriendsDan Rey on Live Streaming [TECHNOLOGYANDFRIENDS 725]
  • LinkedIn Live went from an application process to a self-serve switch. Rey says that when he started, about a year before recording, you had to apply and wait for LinkedIn's approval; now you turn on "creator mode" on your profile, which also unlocks audio-only events, and he believes a profile or company page needs about 150 followers or connections. LinkedIn has no built-in streaming software, so he first tested with StreamYard on a colleague's profile, Mark Steele's, which had access before his own did, and ended up with about a minute of unexplained silent video on Steele's feed.
  • Rey picks his streaming tool on price against the channels he needs. StreamYard's licence limited him to three simultaneous destinations, so within about three months he moved to Restream.io, which allowed unlimited channels for the same price or about $5 a month more; he used it for over a year until a July licensing change made extra channels over twice StreamYard's price, and he went back to StreamYard. He also says he favoured Restream for a long time because a Microsoft viewer told him in a live comment that it runs on Azure while StreamYard runs on AWS.
  • Rey streams to LinkedIn and Twitter, then one topic-dependent third channel. LinkedIn is his main presence and Twitter his second-largest network at over 10,000 followers; the third might be his Tip Tuesday YouTube channel, which he says has no fixed schedule and had last gone live about a month earlier because it earns him nothing. Both StreamYard and Restream run entirely in the browser, and he says the recordings stay viewable after the live event for people who missed it.
  • Streaming public events raises consent and copyright questions. Rey went live from a Sergio Mendes concert and from the DeKalb Corn Classic 10K without asking the mayor, the anthem singer or attendees; he notes some events carry a disclaimer that attendance grants use of your likeness, and says this matters more when streaming for or on behalf of a larger company.
  • A Teams meeting can go live directly using a stream URL and key. Rey demonstrated copying the stream URL and key from LinkedIn's go-live page into a Teams meeting's custom streaming setting, so the meeting is recorded and broadcast at once; on the version of Teams they tried, he concluded it streams to only one destination at a time, though stopping the stream unlocks the fields to switch to another. He also says StreamYard's custom RTMP output can feed a production into a Teams live event.
Andrej KarpathyBuilding makemore Part 3: Activations & Gradients, BatchNorm
synopsis

Staying with the same character-level MLP from the previous lecture rather than moving to recurrent networks as planned, he spends the session diagnosing and fixing exactly what can go wrong inside a neural network's activations and gradients at initialization — pushing validation loss from 2.17 down to about 2.10 through nothing but better initialization and normalization — because, as he argues, these same failure modes become far more dangerous once networks get much deeper, which is the whole reason to understand them now.

The hockey-stick loss. Fresh off training, he notices the very first loss recorded is 27, wildly higher than the ~3.29 a 27-way uniform guess should cost (negative log of 1/27). The cause: default initialization produces logits with large, arbitrary magnitudes, so the softmax comes out confidently wrong for most examples rather than diffuse; the first chunk of training is then spent simply shrinking those logits back toward zero, visible as a sharp early drop in the loss curve before real learning starts. The fix is to initialize the final layer's weights (and biases) near zero, which immediately produces the expected initial loss, removes the hockey-stick shape, and — because training time is no longer wasted correcting an artificial problem — measurably improves the final loss.

Saturated tanh units and dead neurons. A histogram of the hidden layer's activations reveals most values pinned at exactly +1 or -1 — tanh fully saturated. This matters because tanh's backward-pass gradient is (1 − output²): an output near ±1 multiplies the incoming gradient by nearly zero, effectively cutting off learning for that unit on that example. Checking column-by-column for neurons where every example saturates reveals the more serious failure mode: a "dead neuron" that never receives a gradient again and is permanently stuck — a risk he notes is shared by sigmoid (also a squashing function) and ReLU (a dead ReLU, unable to recover once knocked into producing only negative pre-activations, sometimes triggered by too-high a learning rate). The fix mirrors the logit fix: shrink the weights and biases feeding into the hidden layer so pre-activations land in a reasonable range, eliminating the saturated tail and further improving loss.

Principled initialization. Rather than hand-picking scale factors by trial and error, he derives the general rule from Kaiming/He initialization: to keep a layer's output variance equal to its input's, weights should be drawn with standard deviation equal to a gain divided by the square root of fan-in. The gain compensates for how much a given nonlinearity compresses its input distribution — 1 for a purely linear layer, but roughly 5/3 for tanh, whose squashing otherwise causes activation variance to shrink toward zero as it passes through successive layers (demonstrated directly: gain too low and deeper layers' activations collapse toward zero; gain too high and they oversaturate). He's candid that this precise tuning mattered enormously before several later innovations — residual connections, various normalization layers, and better optimizers like Adam — made networks considerably more forgiving of imperfect initialization.

Batch normalization. The core insight, introduced by a 2015 Google paper, is disarmingly direct: since what's wanted is roughly Gaussian pre-activations at every layer, why not just standardize them directly — subtract the batch's mean, divide by its standard deviation — since that operation is fully differentiable? To avoid permanently forcing a fixed distribution, this is paired with a learnable gain and bias so the network can still shift or reshape the distribution as training demands. He implements it as a drop-in layer placed after linear layers and before nonlinearities, demonstrating that it makes networks dramatically more robust to the exact scale of the preceding weights — even removing fan-in scaling entirely still trains reasonably once BatchNorm is present, though the learning rate may need retuning since BatchNorm's own backward pass changes how gradient scale relates to the weights feeding into it.

The catch is a real one: normalizing over the batch means an example's hidden state is no longer a pure function of that example — it now depends on whichever other examples happened to land in the same batch, introducing "jitter" that acts as an accidental regularizer but is also, he says, a persistent source of subtle bugs, which is why later techniques (layer, instance, and group normalization) were developed to decouple examples again. A related practical wrinkle: since BatchNorm re-centers its input, any bias term in the preceding linear layer is mathematically canceled out and wastes a parameter, so linear layers immediately before a BatchNorm should disable their bias. For inference on a single example, where batch statistics don't exist, the layer instead uses a running mean and variance accumulated via exponential moving average during training — the same approach used in PyTorch's own BatchNorm1d, whose parameters (momentum, epsilon, affine, track_running_stats) he walks through directly, noting that overly large momentum can be unstable for small batch sizes.

Diagnostics for a healthy network. Beyond the fixes, he introduces the habit of actively monitoring three things during training: histograms of forward-pass activations and backward-pass gradients across layers, watching for saturation or asymmetric shrink/growth with depth, and — most usefully — the ratio of each parameter's update size to its own data scale, plotted on a log scale over training. His rule of thumb is that this ratio should hover around 10⁻³; consistently far below suggests the learning rate is too low, far above suggests it's too high, and BatchNorm-equipped networks are shown to be considerably more forgiving of these settings than the raw linear-tanh stack was. He closes by noting these gains didn't actually beat the model's previous best loss — because the real bottleneck at this point is a context window of only three characters, not optimization — foreshadowing the shift to recurrent architectures and, eventually, Transformers.

  • Initializing the output layer near zero fixes the "hockey stick" loss. With default init the logits start large and confidently wrong, recording a loss of 27 against an expected ~3.29 (negative log of 1/27 for a uniform guess over the vocabulary); the first hundred-plus training steps are then wasted just shrinking those logits back down rather than learning, so scaling the final layer's weights down fixes both the starting loss and the shape of the loss curve.
  • Saturated tanh units silently kill gradients, and can produce permanently "dead" neurons. Because tanh's local backward gradient is 1 minus its output squared, an activation pinned near +1 or -1 multiplies incoming gradient by nearly zero; if every example in the data ever pushes a given neuron into that flat region, that neuron never receives a gradient again and stops learning entirely — a failure mode he notes also afflicts sigmoid and ReLU (a "dead ReLU").
  • Weights should be scaled by gain divided by the square root of fan-in, where the gain compensates for how much a nonlinearity squashes its input. He derives this from the Kaiming/He initialization paper, showing empirically that a gain of 1 works for a stack of pure linear layers but a tanh sandwich needs roughly 5/3 to counteract tanh's contraction of the distribution — too small a gain and activations shrink toward zero with depth, too large and they saturate, and this fragility is exactly what made very deep networks hard to train before modern fixes existed.
  • BatchNorm stabilizes deep networks by forcibly standardizing activations, but this comes at the cost of coupling examples within a batch. Subtracting the batch's mean and dividing by its standard deviation (then restoring flexibility via a learned gain and bias) removes the need to hand-tune weight scale precisely at every layer; the side effect is that any one example's hidden state now depends on which other examples happen to share its batch, which acts as an accidental regularizer but also introduces the bugs and awkwardness (needing running mean/variance estimates to support single-example inference) that motivated later alternatives like layer normalization.
Crosstalk SolutionsIt's Smaller, but Is It Faster? Testing the tp-link EAP610 v2.
  • The shrunken version 2 slightly outperforms the original in testing. The host concludes TP-Link, with the EAP610 v2, significantly shrank the Wi-Fi 6 access point while improving speeds a little, answering the common complaint that its APs are too large. He doesn't know whether v1 will be end-of-lifed and wants the same treatment for TP-Link's more powerful APs.
  • The new model is cheap, compact and flexibly powered. It lists at $99.99 MSRP, is just over 6 inches across and 1.3 inches tall, with a 4 dBi 2.4 GHz antenna rated up to 574 Mbps and a 5 dBi 5 GHz antenna up to 1.2 Gbps. It runs on 802.3at PoE, 48 V passive PoE or an included 12 V 1.4 A DC adapter, and is managed by an Omada controller (hardware OC200, software or cloud).
  • Averaged runs from three devices all favoured v2 slightly. The host ran five tests per device (his PC, an iPhone 13, an iPad) against an OpenSpeedTest server on Ubuntu 22.04, through a TP-Link TL-SG2210MP switch and TL-R605 router, with identical positioning. He flags that the server's gigabit NIC may have capped results, and that the iPad's uploads ran about half its downloads on both APs.
  • Five-minute sustained stress tests showed the two essentially tied. Run twice per AP from the PC only, v2 led on download by what the host eyeballed as roughly 10 (units unstated) and trailed slightly on upload, which he called splitting hairs.
Crosstalk SolutionsGigabeam test setup - 60GHz PTP!
  • The host's bench test link delivered near-gigabit throughput. The host bridged two GigaBeams across his office, each powered from a US-8-150W switch port set to 24 V passive PoE, with one side feeding a UDM Pro and an OpenSpeedTest server on a Shuttle PC. He measured roughly 712 Mbps down and 941 Mbps up from a wired laptop across the link.
GCastRegistering an Application with Azure Active Directory [GCast 137]
  • Registering an app is the first step to Azure AD authentication. In Azure Active Directory, choose App Registrations and then New Registration; Giard notes you need admin-level permissions on the account. Only two fields are required. The first is a name, and he advises one that is easy to identify and search for later, since the registrations list is found with a filter box.
  • The second required field sets who may authenticate. The options are this tenant only; multiple Active Directory tenants (customers, partners or acquired companies, which needs a trust relationship); multiple tenants plus Microsoft accounts such as Xbox, Skype, Outlook.com, Hotmail and Live; or personal accounts such as Gmail addresses that the directory trusts. Giard picks single tenant and later shows it can be switched to multi-tenant on the registration's details page.
  • The redirect URI is optional and can be set later. It names the page or screen a web, mobile or rich-client app sends the user to after a successful sign-in. Giard leaves it blank because not every app has a user interface: a registration may be for web service calls only.
  • A client secret acts as the registration's password. Giard says a bearer token for a web service call needs one. Under the registration's certificates and secrets, New client secret takes a name and an expiry, from soon to two years or a custom range with a future start date; he sets three months.
  • The secret's value can be copied only once. After a page refresh the value is gone and is not regenerated, so Giard says to copy it immediately to a safe place such as a key vault or a locked-down directory. Anyone holding it can create a token and impersonate the account.
Technology and FriendsCaito Scherr on Apache Flink [TECHNOLOGYANDFRIENDS 724]
  • Flink's exactly-once guarantee scales where other engines' does not. Scherr, developer advocate at Ververica, says Flink's checkpoints and savepoints, delineated by watermarks, let it reread only the section since the last checkpoint without stopping the stream, so nothing is duplicated; she says other engines reread or store data differently and build up backlog. Because small proofs of concept show similar latency and throughput across engines, she urges testing with masses of generated data (DataGen, Faker).
  • Its commonest use is fintech fraud and anomaly detection. Scherr puts that at perhaps "30% or 40% or 50%" of use cases, the rest being unusually varied, because it needs near-real-time results with no duplicates (no double charge, no anomaly flagged twice); machine learning and analytics are also big. Flink is stateful and keeps snapshot metadata locally or externally for rollbacks, but the data itself sits in external databases reached through built-in or third-party connectors.
  • Newcomers should start with Flink SQL, not Java, Scala or PyFlink. She says Flink SQL abstracts much of the complexity, is language agnostic and suits DevOps people, though Java 8 or 11 must still be installed because parts of it use the Java heap. Installation was local at recording, a roughly three-step download she has live-coded in about five minutes or less; she hinted at unannounced changes to that answer.
  • Flink itself is free; Ververica earns from an enterprise platform. Flink is licensed through the Apache Software Foundation like any Apache project, while Ververica, which wrote and maintains it, sells the Ververica Platform for deployment, integration with other tooling, analytics and production support, with a free community edition and paid tiers.
  • The open-source community is what drew her to Flink. At a previous employer new to stream processing, Scherr says a patch she submitted for an odd use case was approved within a couple of days; she calls Flink's mailing list one of the most active in Apache. She recommends meetups, where she says the founders and engineers from Netflix and Apple give talks.
Technology and FriendsSimeon Kakpovi and Greg Schloemer on a Cybersecurity game [TECHNOLOGYANDFRIENDS 723]
  • KC7 generates realistic fake security data so anyone can practise. Kakpovi says learners need realistic data that no one will hand over for privacy and secrecy reasons, so the game simulates a fictitious company's employees sending email and clicking links, 98–99% of it benign, with adversaries mixing in credential-harvesting and phishing links for the player to find.
  • Players pivot from a single tip to an actor's whole pattern. A game starts from a tip, such as a fake tweet naming a suspicious domain, and players weigh how confidently related senders and domains are malicious; Schloemer says modelled actors reuse themed keywords (Viking words like "Norse" in email addresses). All querying is in Azure Data Explorer with KQL, the tools the two use daily as threat analysts, on a free Kusto cluster loaded by a script from their GitHub page.
  • Each game can be regenerated, so answers cannot be memorised. Kakpovi built a small manual dataset almost four years earlier at a professor's request; now a button produces a new company, adversaries and TTPs. The name refers to the seventh, final stage of the Lockheed Martin cyber kill chain, and they intend the full game to cover all seven phases, claiming that anyone who can play it can work in cyber.
  • Business students with no technical background asked how to make it a career. During a Microsoft Hackathon team of over 100 people, they ran the game at Montgomery College in Maryland and a teammate took it to Florida Memorial University, described as South Florida's only HBCU. About an hour to an hour and a half in, a student asked how to do it for a living, and several gave similar feedback.
  • They blame cybersecurity's lack of diversity on access, not skill. Schloemer describes a cycle in which you need data to get the job and the job to get the data; Kakpovi, a Howard University alumnus, recalls a Lockheed Martin Cyber Analyst Challenge with four Black people in a room of 100, himself and three co-workers. KC7 is free and open source, and the two, analysts rather than developers, ask for contributions by pull request.
GCastCreating an Azure Active Directory User [GCast 136]
  • Only an administrator can add users, from the Users blade. Giard logs into the Azure portal as an admin of a Contoso sample directory. He notes Azure Active Directory's overview page has its own "add user" link, but he prefers going to Users in the left menu and choosing "new user".
  • New user offers two paths: create internally or invite externally. "Create new user" makes an account inside this Active Directory. Inviting an external user sends an email that lets someone from elsewhere join.
  • Only the user name and name are required. The user name becomes the sign-in address under the directory's onmicrosoft.com domain and must be unique within the directory. Giard says everything else on the form is optional and can be changed later. The other options he covers are an auto-generated password, which he shows and says to copy; assigning a group or a role, where the default role is User; "block sign in", for someone who starts next week; location; and job details such as title, department, company and manager.
  • Profile properties can be sent as claims in a token. Editing the user after creation shows many more fields the create form never asked for, such as city and address. Giard says this information, along with groups, roles, job title and manager, can be passed to another application in a token so it knows who the person is.
  • The same user page handles password resets and deletion. Giard shows resetting a password, for a user who has forgotten theirs, and deleting a user who has left the company. Deletion asks for a second confirmation.
Technology and FriendsMike Benkovich on Infrastructure as Code [TECHNOLOGYANDFRIENDS 722]
synopsis

Benkovich, an Azure consultant who has used ARM templates for several years and Terraform for about the last nine months, argues that infrastructure should be described in code for repeatable, governed deployments, and that the choice between Azure's own tools and Terraform comes down to how you want state managed and whether you need more than one cloud.

Why code, not the portal. Portal clicks cannot be repeated reliably. Templates take per-environment parameters and carry naming and tagging standards that lower the cost of ownership. Infrastructure is deployed once; the app's code ships many times through its own pipeline.

ARM in Visual Studio and VS Code. Benkovich deploys a web-app-plus-SQL template from a Visual Studio resource group project and moves the admin password into Key Vault. He finds Visual Studio's IntelliSense weak, and prefers VS Code with the ARM extension for building names from a prefix and environment parameters, and for defining outputs passed back to the az deployment group create CLI.

Bicep. Because ARM is verbose, Bicep offers a cleaner language that compiles to ARM; he decompiles his JSON template into Bicep live.

Terraform. It installs via Chocolatey, then runs terraform init, plan and apply, prompting for app name, environment and colour and writing a local .tfstate file. Its resources appear in Azure without a deployment record, since it calls the REST API. Benkovich says that difference in history and state is what should drive the choice.

  • Infrastructure as code buys repeatability and enforceable governance. Benkovich says clicking through the portal is not repeatable, while templates can apply one naming and tagging formula on any cloud or on-premises, and take per-environment parameters, such as large Kubernetes node VMs in production and smaller ones in dev or test.
  • Deploy infrastructure once, and the app's code many times. Benkovich treats the IaC deployment as building the environment the code runs in, with a separate continuous delivery pipeline shipping the code afterwards; Giard adds that in practice it would run from a script or CI/CD tool rather than a right-click in Visual Studio.
  • VS Code beats Visual Studio for authoring ARM templates. Benkovich says Visual Studio's resource group project gives a starter template (web app plus SQL) but weak IntelliSense on parameters, while VS Code with the ARM extension, snippets and multi-line editing handles his concat-based naming variables better. In his demo he swaps a plain-text password in the parameter file for a Key Vault secret.
  • Bicep is a more readable language that compiles to ARM. Benkovich calls it an ARM transpiler: az bicep decompile turned his verbose JSON template into a cleaner Bicep file, and some people liken its format to Terraform's.
  • Choose Bicep or Terraform by how you manage state and clouds. Benkovich says ARM/Bicep are Azure-only and record each deployment in Azure, while Terraform works across Azure, AWS, Google and more and keeps its own .tfstate file; his Terraform-built resources left no entry in the portal's deployment history because it went through the REST API.
Crosstalk SolutionsWhat’s the best option for a router at BestBuy? Let’s find out!
  • At Best Buy, the host would pick Netgear's cheaper Wi-Fi 6 router. Skipping the whole-home mesh kits and anything branded "gaming", he narrows it to two Netgear models with near-identical specs, the AX2400 Nighthawk at $179 and the AX1800 at $139, and buys the AX1800. He passes over a $500 Nighthawk with a 2.5 Gbps Ethernet port, and a Linksys Wi-Fi 6 router at $149 that he says matches another model costing twice as much.
Crosstalk SolutionsExclusive Crosstalk video on Facebook! Facebook.com/crosstalksolutions
  • Crosstalk is posting exclusive content on Facebook, not YouTube. The first one, "My G4 Doorbell Pro broke", goes through the host's troubleshooting steps for getting a UniFi G4 Doorbell Pro back online, and is only on facebook.com/crosstalksolutions. The host adds that the channel is now on Instagram and TikTok too.
Technology and FriendsBrian Gorman on Messaging, Queueing, and Eventing in Azure [TECHNOLOGYANDFRIENDS 721]
synopsis

Gorman, an Azure technical trainer who teaches the AZ-204 exam, ends on one deciding question for Azure's four messaging services: if someone must acknowledge that the work was done, use Service Bus. If not, an event service will do. He warns beginners against reaching first for custom Event Grid events.

Why decouple. Asynchronous messaging lets the UI accept data at any rate and keep giving users feedback, while the back end processes later, overnight if need be. Giard frames this as a staging area that makes the system more scalable. Gorman adds that it is more responsive too.

Events. Gorman separates Event Hub from Event Grid, which can be combined but need not be. Event Hub handles streaming big data at millions of records per minute, with up to 32 partitions, consumer groups that replay independently (so duplicate reads are possible). Event Grid reacts to discrete events such as those Azure Monitor logs, with filtered subscribers.

Messages. A message carries a payload where an event carries only a description. Service Bus queues guarantee FIFO order, and topics give pub-sub, shown in his movie-list demo. Storage Queue is cheaper and simpler, with no guaranteed order, a 60-second lease, a seven-day purge and over 80 GB of capacity.

Choosing. He credits an attendee named Barry with the acknowledgement test, and cites Service Bus dead-letter queues and the portal's message view, which he thinks is still in preview.

  • Queues and hubs decouple the front end from the back end. Gorman says users can submit data as fast as they like without the UI bogging down, while the database scales or stays small and, if processing is not urgent, catches up overnight.
  • Event Hub and Event Grid are separate services, not companions. Gorman says Event Hub ingests big data and streaming telemetry at millions of records per minute, with up to 32 partitions (around four for a typical workload) and consumer groups that can each replay events, which opens the door to duplicate reads. Event Grid responds to one-off events, such as those logged by Azure Monitor, and fans them out to filtered subscribers; IoT Hub is a lighter subset of Event Hub.
  • An event only says something happened; a message carries the payload. Gorman's example is a blob-upload event whose data is just the blob's URL, while a Service Bus message can carry a serialized object. Service Bus queues guarantee first-in-first-out order, and topics give pub-sub, as in his demo that loads 17 or so movies once and serves R-rated, family and all-movies subscribers separately.
  • Storage Queue is the cheaper, lighter option with fewer guarantees. Gorman says it cannot guarantee order, returns an item to the queue after 60 seconds unless the lease is renewed, purges after seven days by default, and holds over 80 GB but with a smaller message size. Service Bus has no automatic purge, though he thinks a time-to-live can be set.
  • Pick Service Bus when you need to know a message was handled. Gorman credits an attendee named Barry at DevUp in St. Louis with the deciding question: does someone need to acknowledge it? Event Hub and Event Grid don't care whether an event was consumed, while Service Bus has dead-letter queues. He also advises beginners not to start with custom Event Grid events, using Storage Queue or Service Bus instead.
Crosstalk SolutionsUber breach update!
  • The host treats the Uber breach as a complete compromise. He advises changing an Uber password now to be safe. The New York Times broke the story first, after the attacker contacted it with images of Uber's internal email, cloud and code repositories.
  • The attacker got in through social engineering and a hard-coded credential. As reported, an 18-year-old who did it "just for fun" social-engineered an employee, logged in over VPN, and scanned the intranet. On a network share he found PowerShell scripts, one with an admin username and password in it, and used those to take over nearly everything. He then announced the breach on Uber's internal Slack, and staff at first took it for a joke.
Scott HanselmanWifi enabled C64 with WiModem talking to a BBS
  • Two hardware "lies" bridge a 40-year-old computer to the internet. Hanselman shows his Commodore 64, his first computer and recently working again. A WiModem on his Wi-Fi tells it that it is an old-style modem, and a cartridge with a Raspberry Pi inside tells it that it is a floppy disk.
  • Dialing a BBS now means connecting to a network address and port. In his terminal program, which he likens to a text-mode web browser, he switches between 40- and 80-character modes and moves from 300 to 2400 bits per second. He then "dials" an address and port (not port 80). He says the board on the other end is another Commodore 64, and it asks whether his machine can do color and graphics.
  • BBSes were social media before social media. Hanselman says people have made bulletin board systems available again. He checks his email on the board (no mail) and presents the demo as nostalgia for the fun people had calling each other's machines by phone.
Crosstalk SolutionsSTIR/SHAKEN and the Death of the Small VoIP Carrier
synopsis

The host explains why STIR/SHAKEN, the FCC's anti-robocall call-signing regime, forced Crosstalk to close Crosstalk SIP, its resold SIP-trunking service. The tokens, certificates and redundant SBC infrastructure now required of any provider that bills end users cost more than the business made. He ends sceptical that the rules will stop robocallers or bind the large carriers.

How it works. Spammers spoof local caller IDs, which open-source tools make easy. STIR/SHAKEN signs each call with a certificate at a session border controller and grades it A, B or C by how much the carrier can verify. Receiving carriers use the grade to judge spam.

The business that died. Crosstalk SIP served 60–70 customers on VoIP Innovations wholesale minutes for about three years, at a 15–20% margin. Its upstream carrier signed the calls for it.

What compliance takes. ClearlyIP CEO Tony Lewis lists the steps: an iconectiv token priced off 499 revenue with a few-thousand-dollar minimum, a certificate costing thousands a year, a sandbox test suite, and geographically redundant SBCs that hold customer DIDs and credentials and handle 911.

The rule change. The host's upstream carrier and a telecom lawyer both said he was covered. The FCC then ruled that whoever bills the end user must sign the calls, and ended the under-100,000-subscriber exemption on July 1. It sent legal letters to firms claiming compliance without a certificate, and the robocall mitigation database now requires full STIR/SHAKEN.

Where it falls short. Lewis says most T-Mobile "likely spam" calls arrive at C, partly because SS7 legs strip the signing. Non-IP networks must adopt a variant next year. The host calls the result a gold rush for consolidators that squeezes small providers while well-lobbied carriers and offshore robocallers carry on.

  • STIR/SHAKEN made Crosstalk close its own SIP-trunking business. Crosstalk SIP ran for about three years and served 60–70 customers, peaking at 130,000–140,000 calls a month. It resold wholesale minutes from VoIP Innovations, which Sangoma later bought, at a 15–20% margin after billing, tax and staff costs. All customers have now been moved to other providers.
  • Attestation grades how far a carrier vouches for a caller. Calls are signed with a digital certificate at a session border controller (SBC). Level A means the carrier knows the caller and that they own the number, B means it knows the caller but not the number (for example Follow Me forwarding), and C is a gateway call with neither verified, which is the grade most likely flagged as spam. The host cites Bandwidth.com: 3–5 billion robocalls a month, about 40% thought to be fraud.
  • Compliance costs a small reseller more than its margin can carry. According to ClearlyIP CEO Tony Lewis, a carrier needs a token from iconectiv, priced as a share of its FCC 499 filing revenue with a minimum of a few thousand dollars a year. It then needs a certificate costing thousands a year, a sandbox test suite, and geographically redundant SBCs that handle 911. Lewis calls the cheaper option, call-forking signing services that need a PJSIP patch, convoluted.
  • The host's advisers wrongly told him a reseller was covered. Both VoIP Innovations and a telecom lawyer said its compliance covered him. Around June–July the FCC clarified that whoever bills the end user is the originating provider and must sign the calls. The exemption for providers under 100,000 subscribers ended July 1, and the FCC legal department sent warning letters to firms claiming compliance without a certificate. Lewis says Inteliquent will now sign uncertified customers' calls only at C.
  • Lewis and the host doubt the system stops robocalls yet. Lewis reports that T-Mobile users see "likely spam" and nine in ten test calls arrive at C, partly because SS7 legs strip SIP headers. The host predicts the large carriers will lobby their way past enforcement while robocallers work around the rules. Disclosure: the two co-run a paid open-source PBX training course.
Andrej KarpathyBuilding makemore Part 2: MLP
synopsis

Moving past the bigram model's fundamental limit — its lookup table grows exponentially with context length, making anything beyond a single prior character intractable — this lecture implements a multi-layer perceptron character-level language model following Bengio et al. 2003, ending with a network that takes three characters of context, embeds them, and predicts the fourth, reaching a validation loss of about 2.17 (versus 2.45 for the bigram model) and producing noticeably more name-like samples like "ham," "joes," and "lela."

The architecture. Each of 27 possible characters is mapped to a point in a low-dimensional embedding space via a lookup table C, initialized randomly and tuned through backpropagation alongside the rest of the network. He demonstrates that indexing into this table is mathematically equivalent to multiplying a one-hot vector by C — framing the embedding step as just a linear layer with no nonlinearity — before settling on the much faster direct-indexing approach. Three character embeddings are concatenated (efficiently, via .view() rather than torch.cat, since reshaping a tensor's view requires no new memory) and fed through a hidden layer with a tanh nonlinearity, then a final linear layer producing 27 logits. The motivating idea from the paper: since embeddings place similar characters or words near each other in this learned space, the model can generalize to combinations it never saw in training — if it learns "a" and "the" behave similarly, it can transfer what it learned about "the dog was running" to "a dog was running" even without that exact phrase in the data.

Practical training mechanics. He switches the hand-rolled softmax-and-negative-log-likelihood computation to PyTorch's F.cross_entropy, noting it's not just more convenient but meaningfully better: it fuses operations into a faster forward and a mathematically simpler backward pass, and critically avoids numerical overflow by internally subtracting the batch's maximum logit before exponentiating — since softmax output is invariant to a constant offset, this keeps very large logits from producing inf. Training then moves to mini-batches of 32 random examples rather than the full 228,000, trading a noisier, approximate gradient for dramatically more update steps in the same time — which turns out to converge faster overall since gradient direction, even if imperfect, is still useful.

Finding a learning rate. Rather than guessing, he sweeps candidate learning rates exponentially from 0.001 to 1 across a short run and plots the resulting loss against the learning rate's exponent, producing a characteristic valley: too low and the loss barely moves, too high and training becomes unstable. The middle of that valley — around 0.1 — gives a defensible starting point, later followed by a manual 10x decay once the loss plateaus, a two-stage schedule he's candid is "janky" compared to production practice but effective enough here.

Diagnosing underfitting versus overfitting. Splitting the data 80/10/10 into train, dev (validation), and test sets, he shows that when training and dev loss track closely (~2.3 each) but remain fairly high, the model is underfitting — too small to be memorizing, let alone generalizing well — and needs more capacity rather than regularization. Tripling the hidden layer's neuron count barely moved the needle, but increasing the character embedding dimensionality from 2 to 10 (with the hidden layer trimmed back to 200 units) dropped the loss meaningfully further, revealing that the 2-dimensional embedding space, not the hidden layer, was the actual bottleneck restricting how much distinct information the network could represent per character. Visualizing the trained 2D embeddings before scaling them up makes this concrete: vowels cluster tightly together, ordinary consonants occupy a shared middle region, and outliers like "q" and the special "." token sit far off on their own — visible evidence that the space is encoding real structure rather than sitting at its random initialization. The chapter closes by explicitly flagging the knobs left untuned — embedding size, hidden layer width, context length, batch size, and learning rate schedule — as an open invitation to keep improving past the 2.17 validation loss reached here.

  • Looking up a row in an embedding table is mathematically identical to multiplying a one-hot vector by that table, framing embeddings as just a linear layer with no nonlinearity. Following Bengio et al. 2003, characters (or words) are mapped into a continuous space that back-propagation shapes during training, which is what lets a model generalize to an unseen phrase like "a dog was running in a ___" if it has learned that "a" and "the" occupy nearby points — a transfer that a pure counting-based model has no way to make.
  • A cheap way to find a good learning rate is to sweep it exponentially across many steps and plot loss against the exponent. Testing candidate rates from 0.001 to 1 revealed a "valley" shape — too low barely moves the loss, too high makes it unstable — with roughly 0.1 identified as the sweet spot for this network; this beat the earlier bigram model's 2.45 loss, reaching about 2.3 with mini-batch gradient descent, which trades a noisier, approximate gradient (computed on random batches of 32 rather than all 228,000 examples) for far more update steps per second.
  • Comparing training loss to held-out validation loss diagnoses whether a model is underfitting or overfitting, and pinpoints the real bottleneck. When both losses were roughly equal and mediocre (~2.3), enlarging the hidden layer from 100 to 300 neurons barely helped, but increasing the embedding dimension from 2 to 10 did — revealing that the 2-dimensional character embeddings, not the hidden layer, were the actual constraint; visualizing the trained 2D embeddings beforehand showed vowels clustered together while rare characters like "q" sat in their own isolated region, confirming the embedding space itself was carrying learned structure.
Technology and FriendsJoe Guadagno on Leadership [TECHNOLOGYANDFRIENDS 720]
  • Leaders grow people; managers direct tasks. Guadagno, a director over several teams at Rocket Mortgage, says the word "manager" is frowned upon there; he contrasts a meme of a manager in a cart whipping staff forward with a leader in front helping pull, and a manager assigning a report by a date with a leader growing the person.
  • Leadership can partly be taught but rests on traits. He names compassion, wanting to grow and putting others first, and ties his own to having trained as a high school math teacher and a philosophy of leaving the world better than he found it.
  • He hires for fit with the team over top technical skill. Across his five teams he weighs communication against engineering skill, and rather than turning everyone into a leader he tries to grow each person toward what they want to be "when they grow up"; he spots leadership potential in those who raise flags and ask what-if questions.
  • Knowing each person individually is why he says turnover stays low. He holds one-on-ones, bans calling people "resources", and tailors how he talks to and praises each one, from a quick Teams thank-you to a public award. Of 55 people over four years, he says about two have quit, against an industry where people change jobs every two or three years; Giard adds that the empathy has to be sincere.
  • Humour eases pressure during outages and deadlines. His teams are known for bad jokes and did costume and Zoom-background days after going remote; he deliberately throws in a joke when an outage has people circling.
Technology and FriendsDon Miller on Headless Content Management Systems [TECHNOLOGYANDFRIENDS 719]
  • Headless CMS separates content editing from a custom-built front end. Miller says his agency uses WordPress only as a database and admin portal for non-developers, pulls the content via GraphQL or the REST API, and renders it with Next.js into static pages hosted on Netlify behind Cloudflare. The page is rebuilt when a post is saved, so no database call happens when a visitor opens it.
  • Google's page-speed demands are what make plain WordPress a problem. Miller says Google's 2021 algorithm change around Core Web Vitals wants pages to load on mobile in about a second. A WYSIWYG builder like Elementor is convenient for a quick personal site, but its generated code makes that hard for bigger companies that depend on SEO or paid advertising.
  • Moving the front end into code brings a real development lifecycle. Miller says changes now live in GitHub and go through pull requests, so a breakage can be traced back and rolled back. Netlify builds a preview URL for each pull request for clients to approve before merging, and A/B testing is possible but rarely done because it is a lot of work.
  • Serving cached static copies improves security and scalability. Miller says Cloudflare copies the rendered site to locations worldwide, so an attacker, including in a DDoS attack, hits a static copy with no server or database behind it. He stresses he would never call it unhackable.
  • Headless takes developers, not WordPress developers. Miller says retraining his WordPress staff from CSS and PHP into JavaScript front-end work was one of the largest challenges. He knows of no prepackaged offering that avoids custom development, and says that at his talks four of five attendees don't know what headless is.
Andrej KarpathyThe spelled-out intro to language modeling: building makemore
synopsis

This lecture builds the first and simplest member of the makemore family — a bigram, character-level name generator — twice over: once by directly counting character-pair frequencies, and once by training a single-layer neural network with gradient descent, arriving at the identical ~2.45 loss and identical output samples both ways. The destination isn't a strong model (bigram statistics are, in his words, "just really terrible" — producing garbled near-names rather than good ones) but a fully worked bridge between classical counting-based statistical modeling and the gradient-based neural framework that every later lecture in the series builds on.

Counting bigrams. Starting from a 32,000-name dataset, he treats each name as a sequence of bigrams — consecutive character pairs — wrapping every word with a special start/end token (later unified into a single "." token) so the model also learns which characters are likely to begin or end a name. Every one of the roughly 27x27 possible pairs (26 letters plus the special token) gets a count, tallied into a PyTorch tensor and visualized as a heatmap, revealing structure like "N" being an extremely common ending character.

From counts to a generative model. Normalizing each row of the count matrix into a probability distribution turns raw statistics into something samplable: starting from the special start token, torch.multinomial draws a next character according to that row's probabilities, then repeats using the new character's row, until the end token is drawn. This produces recognizably name-adjacent gibberish ("Keanu," "O'Reilly"-ish fragments), which he confirms is genuinely better than a uniform baseline, even though it's still clearly a weak model since it only ever looks one character back.

Measuring model quality and its sharp edges. To turn "how good is this model" into a single number, he introduces the likelihood (the product of the probabilities the model assigns to every actual next character in the training data) and its practical cousins: log-likelihood (summing logs instead of multiplying tiny numbers) and negative log-likelihood, averaged, which becomes the loss to minimize — lower is better, zero is perfect. A sharp edge appears immediately: querying the model on a plausible but unseen bigram (a name ending in "...jq") returns exactly zero probability, and log(0) is negative infinity, breaking the loss entirely. The fix, "model smoothing," adds a small fake count to every bigram before normalizing, guaranteeing no probability is ever exactly zero — trading a little sharpness for robustness.

A broadcasting bug as a teaching moment. Converting the row-normalization step into an efficient, vectorized operation exposes a real and easy-to-miss PyTorch pitfall: summing across a dimension with keepdim=False collapses a column-shaped sum into a plain vector, which broadcasting rules then silently reinterpret as a row vector when it's used in a division — replicating it in the wrong direction and normalizing columns instead of rows. The code runs without any error either way; only checking that each row of the result actually sums to one reveals which version is correct. He treats this less as an aside than as a warning: broadcasting semantics deserve real study, because they fail silently rather than loudly.

Reframing bigram modeling as a neural network. Rather than storing probabilities in a table, he encodes each input character as a one-hot vector and feeds it through a single linear layer of 27 neurons with no bias and no nonlinearity, producing 27 raw outputs per input that he explicitly interprets as log-counts ("logits"). Exponentiating these gives count-like positive numbers, and normalizing them (the two steps together forming a softmax) yields the same shape of probability distribution as the counting approach — and because a one-hot vector times a weight matrix simply extracts one row of that matrix, the trained weights end up being nothing more than the log of the original count table, arrived at from a different direction. Training proceeds exactly as with micrograd: a forward pass computes probabilities and the negative-log-likelihood loss, loss.backward() populates gradients via PyTorch's autograd, and a simple update step nudges every weight against its gradient — with gradient descent alone converging to the same ~2.45 loss the explicit counting approach found directly, confirming the two are mathematically the same model reached by different roads.

Why the neural framework is worth the extra machinery. Counting works perfectly well for a single character of context because a 27x27 table is small and every cell gets plenty of data, but scaling to two, three, or ten characters of context blows the table up exponentially and starves most entries of data — a wall that counting fundamentally can't get past. The gradient-based approach doesn't have this ceiling: nothing about the loss function, the softmax, or the optimization changes as the input representation grows more complex, which is exactly what lets the same machinery scale, in later lectures, from a single input character all the way to full Transformer architectures. A closing observation ties the two approaches together further: penalizing large weights in the loss (an L2 regularization term) pushes weights toward zero, which flattens the softmax output toward uniform — mathematically the same effect as adding fake counts in the table-based version, just implemented as a tunable loss term instead of a table edit.

  • Broadcasting silently normalized the wrong axis when a dimension was dropped instead of kept. Dividing a 27x27 count matrix by its row sums gave correct results only with keepdim=True; without it, the summed vector's shape collapsed from a column to a bare 27-length vector, which broadcasting then reinterpreted as a row vector — replicating it vertically and normalizing columns instead of rows, a bug that produced no error and only surfaced by checking that each row actually summed to one.
  • A model that assigns exactly zero probability to an unseen combination breaks the entire loss function, since log(0) is negative infinity. Testing the trained bigram model on a plausible-looking but unseen name fragment produced infinite loss because that particular character pair never occurred in training; the fix, "model smoothing," adds fake counts (e.g., +1) to every bigram before normalizing, trading a bit of accuracy for guaranteed non-zero probabilities everywhere.
  • A single-layer neural network trained with gradient descent converges to the exact same model as directly counting bigram frequencies. The network's output logits are interpreted as log-counts, exponentiated into count-like values, then normalized via softmax into probabilities; because a one-hot input vector times the weight matrix simply plucks out one row of that matrix, the trained weights end up mathematically equivalent to the log of the explicit count table — reaching the identical ~2.45 loss and identical samples, but via a framework that generalizes to arbitrarily more complex networks (all the way to transformers) where explicit counting is no longer feasible.
  • Penalizing large weights during gradient-based training is mathematically equivalent to count-based smoothing. Adding a term that grows with the sum of the weights squared pushes weights toward zero, which — because zero weights produce equal logits and therefore a uniform softmax output — has the same flattening effect on predicted probabilities as adding fake counts to a frequency table, just implemented as a regularization term in the loss instead of a table edit.
Crosstalk SolutionsThe Best Access Point Placement Tips!
  • Map the walls before placing any access point. The host traces the floor plan in the UniFi Design Center (design.ui.com), marking exterior walls, interior walls and window or door openings. Exterior walls weaken signal more because of stucco or siding, so knowing which walls are exterior matters; he had to guess for the garage and closet walls.
  • Two U6 Lites cover a 1,575 sq ft house. The host puts the $99 access points in two spots. One goes in the dining room near the back window, to reach the patio and master bedroom. The other goes just outside the opening between bedrooms 2 and 3 rather than inside their hallway, which keeps the foyer and garage in range. His target is light blue coverage everywhere at 2.4 GHz, with about –60 dBm (green) at worst.
  • Too much overlap makes phones flip-flop between access points. iPhones are especially aggressive about jumping to the strongest one. The host's fix is to keep the strongest zones barely touching and to lower output power on the dining-room access point, which serves an open space with few obstructions. He calls tuning more art than science: test and adjust.
  • 5 GHz leaves weak spots that 2.4 GHz covers. On 5 GHz the far bedroom and the garage are weaker, but most devices drop to 2.4 GHz and keep working. If the garage needs strong Wi-Fi, the host suggests adding an access point there at low power so it covers only the garage.
  • Design Center plans transfer to other brands' access points. The host says the results hold for any brand whose signal strength is similar to the Ubiquiti model placed in the plan.
Polis PanditProgramming Note: Pausing PolisPandit
  • This post includes paid content that is not available publicly. See the original source.