Souk's Notes - Unofficial Summaries

2025-07-22 · open the original ↗

• LLMs lack the cross-domain learning and values that general intelligence needs. Bronfman, CEO and co-founder of Pecan AI with PhDs in computational neuroscience and philosophy, defines intelligence as solving problems of similar complexity across domains in ways that create value for the solver. He sees no conceptual reason to call today's models general, any more than 1990s Deep Blue. Under his and colleagues' contested unlimited associative learning theory, such transfer requires a unified value system, and LLMs "don't care about anything" beyond an engineer-chosen loss.
• The next leap needs different architectures and embodiment, not scale. Bronfman doubts that bigger or longer-context models are the next qualitative leap. He says neural networks mimic only a small fraction of known brain dynamics, and he favours architectures with layers of meta-learning that modulate the network. Embodiment, he argues, would widen what can be learned and could supply real value rather than a loss function. He adds that machine intelligence need not copy the brain, and that AGI is still "quite far" with existing technology, though he declines to give a date.
• Generative AI shows combinatorial creativity, not transformational creativity. Using Margaret Boden's distinction, Bronfman says LLMs can assemble existing pieces, such as jokes from several comedians, into something new. Transformational creativity takes a wholly new perspective, as with Einstein recasting gravity as geometry, and he argues it is very hard to reach with domain-specific intelligence.
• Prediction, not generation, still drives business value. Bronfman stands by his keynote claim from two years earlier that "prediction is all you need" for businesses. He credits machine learning and predictive modelling for the rise of Google, Facebook, Amazon, Uber and Spotify, and says business value in 99% of cases relies on prediction or advanced analysis. He estimates machine learning has uncovered perhaps 5% of its potential in companies.
• Business framing and data transformation matter more than accuracy. Bronfman says a 91% versus 91.5% AUC gap is usually meaningless. A churn model at 95% accuracy with 7 days' notice may be worth less than 70% at 14 days. He says every company's data fingerprint makes transforming data into model-ready form the real barrier. Pecan's conversational copilot targets that barrier, and he argues it frees data scientists for the nuanced cases rather than replacing them.

2025-07-21 · open the original ↗

• The host warns VoIP resellers not to become 3CX partners. He has never used the product and concedes it may be good, but says he would never rely on 3CX for his customers because of how it treats channel partners; once a partner is cut off, he says, the customers they brought in become 3CX's.
• A former partner reports being dropped for commenting on a competitor. In a LinkedIn post the host reads, the partner says 3CX stripped their partner status within 8 hours of a comment about a rival platform, then removed their help-forum posting rights. The host says he has heard dozens of similar stories.
• The host cites past evidence of partner-hostile terms. He quotes a former 3CX director of sales in a 2018 interview saying partners had to hand over full end-user contact details to get free updates, which 3CX could use to go to customers directly; and in 2023, he says, 3CX ended updates for perpetual licences past a certain version unless customers moved to recurring licences.

2025-07-21 · open the original ↗

• Off-the-shelf guardrail models protect against the wrong risks for regulated domains. Gehrmann, first author of the April arXiv paper "Understanding and Mitigating Risks of Generative AI in Financial Services", treats capital markets and asset management as a case study. He says foundation models are not trained on financial corpora, and safeguard models such as Llama Guard and ShieldGemma encode categories meant for broad, productivity-chatbot populations. Finance, healthcare and law carry jurisdiction-specific obligations, including whether a disclaimer is enough or a question must be blocked outright.
• Evaluate the system in the context where it is deployed. Gehrmann's recommendations: start from general taxonomies such as the NIST AI risk management framework or MLCommons and adapt them to the domain, so compliance teams can categorize risks. Run red-teaming events and count how often the system broke, gave financial advice or refused, which quantifies the risk surface. Don't assume vendors' benchmark scores carry over to downstream applications.

2025-07-20 · open the original ↗

• Longer context makes models more likely to drop their safety alignment. Gehrmann says his paper found that the more context is supplied, the more likely a model is to forget its built-in guardrails. That should not happen in RAG, where the added material is innocuous, and he argues that technically supporting long attention is not the same as being trained to handle long inputs.
• How much context to retrieve is a design trade-off with safety consequences. Gehrmann describes multi-stage retrieval: query parsing and metadata filters, a first pass from hundreds of millions or billions of documents down to a handful, then a costlier re-ranking step. Less context with the answer in it makes the model's job easier, but it turns the problem into precise retrieval. Whole documents give richer context, such as the document type or what the executive summary said. Each choice affects both helpfulness and harmlessness, which he says needs much more research.

2025-07-19 · open the original ↗

• RAG improves honesty but does not make a system more harmless. Gehrmann uses Anthropic's three H's (helpful, honest, harmless) and says hallucination belongs with honesty, where RAG's grounding is a clear gain. RAG is therefore necessary, not dangerous, but harmlessness is a separate question: the same analyst tool could be asked "who is the worst broker and who should I exploit". Built-in model safeguards often break down in RAG setups.
• Grounding enables transparent attribution of every claim. Gehrmann's definition: every figure or statement in an answer, such as a stock price or an analyst's quote, should trace back to a document or the query that produced it. He says this can partly help with harmfulness too, though it is a separate topic.
• Generic guardrails fail on domain risks, so build your own around retrieval. Gehrmann's team released a financial-services content risk taxonomy with 12 categories, with around 100 test queries per category, such as financial misconduct, financial advice, personal information and defamation. They found that open-source guardrails such as Llama Guard, ShieldGemma and NVIDIA's Aegis also fail in that domain. His alternative pipeline is guardrail, retrieval, answer, guardrail, with the whole end-to-end system evaluated by subject-matter experts rather than judged by public benchmark scores.

2025-07-19 · open the original ↗

• This post includes paid content that is not available publicly. See the original source.

2025-07-18 · open the original ↗

• Training data, not architecture choice, decides computer-vision performance. Corso, a University of Michigan professor and co-founder and chief science officer of Voxel51, says his lab noticed that with roughly half a dozen off-the-shelf architectures for a task such as pedestrian detection, results depended more on the dataset than on which model was chosen. Hence Voxel51's mantra, "better data, better models", and his view that data, model and ops are inseparable.
• Rare events make data coverage the hardest part of high-reliability systems. Corso puts US-recorded accidents at roughly one per tens of millions of miles driven. Getting such cases, labeling them and finding failure modes to target with more data is the bottleneck for building 99.999%-accurate systems.
• Datasets outgrew human inspection, which is why Voxel51 builds data tooling. Corso went from a grad-school dataset of about 100 images to ImageNet and then to roughly five billion samples in sets such as LAION-5B. Voxel51's open-source tool, released in 2019 or 2020 (he isn't sure which), has about three million installs, and he says it is aimed at highly technical data scientists.
• Verified auto-labeling lets teams accept about 70% of labels automatically. Corso describes the product, in alpha with customers at the time of recording, as follows: a battery of foundation models labels raw media, and Voxel51's own models rank the outputs so that about 70% can be accepted in batch. The remaining 30% are ranked again so that human reviewers spend time only on corner cases. He expects it to work better for common classes such as pedestrians than for teddy bears or coffee mugs.
• "Curation is the new annotation", and agents will ask humans next. Corso's framing: annotation 1.0 sent everything to humans, and perhaps 1–10% of it proved useful. In 1.5, foundation models pre-filter. His forecast for 2.0, which he says is not yet built, is agentic systems that take a problem statement and unlabeled data and ask humans questions only when necessary.

2025-07-18 · open the original ↗

• Learn machine organization, not necessarily to write assembly. Hyde, author of The Art of Assembly Language, says the average Java programmer needs to understand the machine's low-level behaviour but will never write assembly; only those writing things like compiler libraries must. He traces the view to his own struggles with strings and arrays in Pascal. Hanselman offers knowing stick shift in a world of Ubers. Hyde extends it with nearly forgetting to brake in his old car after an 8,000-mile trip on lane-assist.
• Hand-written assembly's speed edge over compilers has nearly vanished. He puts assembly at 100–1,000 times faster than Applesoft BASIC on the Apple II, 10–100 times faster than 1980s compilers and about 10 times in the 1990s. Today he would struggle to beat a good C compiler by two times. His newer books therefore have a C main program calling assembly functions. The exception he cites is a float-formatting routine 5–10 times faster than the C library, needed on a slow embedded ARM chip.
• Hardware will no longer rescue slow code. Hanselman reads Hyde's Write Great Code warning that performance no longer doubles every couple of years, and Hyde says that will always be true. Speed now comes only from more cores, and he says that 8–16 cores is already past diminishing returns, with the average application topping out around three or four.
• AI-generated code only helps those who already know the subject. Hyde says ChatGPT is sometimes amazing and sometimes "brain dead," and fuller requirements eventually get him what he wants. His real fear is not knowing whose copyright the output tramples. Hanselman counters that Copilot filters copied code and can link its GitHub source, and says he uses AI as a patient Socratic tutor.
• ARM is miserable at the instruction level and broke compatibility. Hyde says he hates RISC encodings and that ARM's 64-bit move was effectively a new instruction set, whereas x86 code has run unchanged since the 386. He calls MASM far better than the GNU assembler, which he says was built to assemble GCC output rather than for people to write by hand.

2025-07-18 · open the original ↗

• Every API is a tool an AI can call. Rabbah, head of product for Postman Flows, argues that LLMs only act on the world through APIs. Sending a message or taking a payment is an API call, and he says the Postman API network holds over 100,000 of them. He credits MCP's popularity to making that connection easier.
• Tools handed to an LLM need documentation and guardrails. Hanselman likens a model with badly built MCP servers, some carrying 85 tools and some destructive, to fumbling in the dark with screwdrivers. Rabbah agrees that a poorly described tool makes the model guess. He argues good descriptions, parameters, authentication and audit trails now matter more, comparing an agent to a child handed powerful tools.
• Visual workflows beat piles of generated code for maintenance. Rabbah says a vibe-coder cannot reason about generated JavaScript, but can check a boxes-and-arrows flow and tweak an edge instead of re-prompting. This is a claim for his own product. He says Flows compiles to a runtime specification on a WebAssembly-based serverless runtime that deploys to any cloud in about a second.
• Resisting higher abstraction misreads where expertise goes. Rabbah recalls customers who wanted to edit the Verilog his Java-like hardware language generated, which he never understood. He sees today's pushback as job security more than nostalgia. He says seniors wield AI well while juniors need coaching. Hanselman, who frames the practice as specification-driven development, says he vibe-coded a sponsor-management app in 42 minutes and never saw its code.
• AI-assisted fixing turns quarters of bug work into weeks. Rabbah says he times each fix from opening Cursor to opening the pull request, and that the team can close 150 customer bugs in a month instead of half a quarter. Flows agents triage feedback and open fixes for an expert to judge. He warns this must not become noise engineers ignore, or replace the thinking people enjoy.

2025-07-17 · open the original ↗

• This post includes paid content that is not available publicly. See the original source.

2025-07-17 · open the original ↗

• RAG exists because a trained model is frozen at its knowledge cutoff. The host describes the setup: a law firm's millions of contracts are searched in natural language, and about half a dozen to a dozen of the most relevant are put into the LLM's context to answer from. The guest adds the rationale: anything after the cutoff (his example is January 2021) can only reach the model through its context, so researchers coupled a search system to a model that is good at synthesizing, summarizing and extracting.
• "RAG" now names any grounding of a response in retrieved data. The guest says search engines already offered answer snippets, a kind of search-enhanced question answering, before the RAG paper. The term now covers structured or unstructured data from one or many sources, not just the vanilla retrieve-then-generate pipeline, which is why he calls it one of the predominant ways applications are built.

2025-07-16 · open the original ↗

• Good data science is judged by the decision it moves. Siddharth Ranganathan, a director of data science at Microsoft, defines it as starting from the right question, being methodologically sound without chasing complexity, and balancing speed with substance; the example is a hurricane-era test of whether marketing campaigns lost impact, answered directionally in 3 or 4 days and followed later by the full analysis. A co-host adds that a six-month causal model may never prove causation, so quick hypotheses and experiments often serve better.
• Set rigor by the cost of being wrong. Ranganathan starts from the decision the analysis informs, defines "good enough" up front, and offers stakeholders a choice, such as a fast directional answer or a two-week deep dive, rather than slowing work unilaterally in the name of rigor. The trade-off checklist taught to the team is speed versus rigor, simplicity versus accuracy, a local versus a global problem, short versus long term, and quantifying impact before the work rather than after; the rule against cherry-picked metrics, biased experiments and misleading visuals stays fixed.
• Being strategic means framing questions, not answering them. Ranganathan's test is outcomes over outputs: an 86% prediction accuracy or an XGBoost model at 80% on a resume says nothing without what changed, a point drawn from hiring as the team grew from 4 to 25 this year. A team that says yes to every request should rank work by which OKR it moves, how far and how fast, check whether engineering has bandwidth to use it, and park the rest for a later quarter.
• Treat a promotion as applying for a new job. Ranganathan's list of traps is staying in execution mode, mistaking activity for impact, over-indexing on technical depth, working solo, weak communication of impact and coasting. The prescription is to show impact above the current level, line up patrons (perhaps three at the target level), meet them on their OKRs, and give the manager, described as "your referral", a ready case; visibility is "a rainbow" of forums, such as the monthly business reviews where Ranganathan has team members present to senior leadership. A co-host adds to ask what readiness requires two or three cycles early, and to rehearse a "road show" from small audiences upward to gather the likely questions.
• Ranganathan predicts role-based agents within a year or two. These would be agents acting as a marketer or a capacity planner, with tech and business skills merging over the longer run; for product data scientists today, an AI feature is measured like any other feature on engagement and retention, while applied scientists need deeper agent and MCP knowledge. The underrated skill named at the close is breaking down an ambiguous problem before solving it.

2025-07-16 · open the original ↗

• This post includes paid content that is not available publicly. See the original source.

2025-07-16 · open the original ↗

• The Dream Router 7 is the simplest entry, but it pins your access point. At $279 it combines gateway, four-port switch with one PoE port, Wi-Fi 7 access point and full UniFi OS apps, and handles multi-gig internet; the host's "mother-in-law router" suits apartments and small homes. Its bundled 64 GB microSD is good for maybe one camera and not mission-critical, and the Wi-Fi sits wherever the ISP hands off.
• The UCG Ultra suits only sub-gigabit lines and network-only use. At $129 it manages up to 30 UniFi devices and over 300 clients with a 2.5 Gb WAN and four 1 Gb LAN ports, but runs only UniFi Network and is rated for 1 Gbps with intrusion prevention. The host would step up to the $199 Cloud Gateway Max, rated 2.3 Gbps, for faster internet or other UniFi apps.
• The Lite 8 PoE switch is optional for mostly-wireless homes. The $109 switch has eight gigabit ports, four with PoE+, for wired devices and access points; without many wired devices a PoE injector for the access point will do.
• The host would pay extra for the U6+ over the U7 Lite. The $30 buys better range: the U6+ ($129) is rated for 1,500 sq ft and about 300 clients against 1,250 sq ft and 200 for the $99 U7 Lite, and on a gigabit switch Wi-Fi 7 gives no noticeable speed gain. Gateway, switch and U6+ total $367, with each piece placed where it works best.
• Cheaper UniFi gear is not the best value for home use. The host rejects the $190 Express 7 as too underpowered and the $89 UK Ultra, a Wi-Fi 5 unit, as only a supplemental backyard access point. He notes most out-of-production UniFi gear on eBay or Facebook Marketplace is still supported.

2025-07-16 · open the original ↗

• Innocuous retrieved documents can circumvent LLMs' built-in safety refusals. Gehrmann describes his team's paper, run by an intern under Bloomberg's responsible-AI research. It paired unsafe queries such as "how do I do insider trading?" with entirely harmless Wikipedia documents. Most models refused the bare query, but with the documents attached their responses were often unsafe, contrary to the common view that RAG's grounding makes systems safer.
• RAG remains unavoidable for grounded answers in finance. Gehrmann says hundreds of billions of pieces of data enter Bloomberg's systems daily, so RAG or similar document-grounded technology is the only way to answer from trusted sources. That is why he says regulated clients need misuse, whether deliberate or accidental, to be identified, blocked and monitored.

2025-07-15 · open the original ↗

• The SunFounder Pironman 5 Max's standout is its dual M.2 riser. It takes two NVMe drives in RAID 1 or RAID 0, or, as in the host's build, one NVMe plus the Raspberry Pi AI Kit, whose Hailo-8L gives about 13 TOPS and which he calls a learning toy. The case also has a tower cooler, RGB fans, a 1-inch stats OLED, exposed GPIO and a board converting micro HDMI to full HDMI.
• The host's build turns the Pi into an offline knowledge store. He added a 1 TB Kingston NVMe drive and installed Internet-in-a-Box, carrying local Wikipedia, over 60,000 Project Gutenberg ebooks and Khan Academy Lite.
• The host rates it highly, with noisy fans the main complaint. He says assembly took about 90 minutes with easy instructions, and he would swap the fans for quieter ones on his own desk. The base model, without AI kit or storage, is about $108 US.

2025-07-15 · open the original ↗

• RAG can strip away a model's built-in safety, even with harmless documents. Gehrmann, head of responsible AI at Bloomberg, describes the paper "RAG LLMs Are Not Safer". Unsafe queries such as "how do I do insider trading?" were paired with innocuous Wikipedia documents, and most models that refused the bare query often gave unsafe answers with the documents attached. He says the more context is added, the more likely models are to forget their alignment, because they are secured for non-RAG setups.
• RAG helps honesty, not harmlessness, so the two need separate evaluation. Using Anthropic's helpful-honest-harmless framing, Gehrmann says RAG reduces hallucination and enables transparent attribution of every figure or quote to a source. That makes it necessary, but not necessarily safer. Harmlessness risks tend to be shared across a sector, and he lists three attack vectors: harmful data in the store, an unsafe model, and malicious users.
• Long context and full LLM search cost too much latency and money to replace retrieval. Gehrmann says needle-in-a-haystack scores don't show a model can handle research-heavy tasks such as Bloomberg's newly released tool searching over 400 million documents. Commercial systems still use 1980s–90s keyword retrieval before semantic search because it is cheap, and users may prefer a fast answer to one five points more accurate.
• Safer base models resist RAG attacks better, but every model broke. Gehrmann says that in their tests Llama 70B and 7B were relatively safer, with 70B a little better. Parameter counts mislead with mixture-of-experts models. Gemma 7B's low unsafe-output rate came largely from refusals, which he says is worthless if it also refuses safe questions, so safety must be paired with helpfulness evaluation.
• Build domain guardrails and evaluate the deployed system in context. Gehrmann's second paper, on generative AI risk in financial services, found that Llama Guard, ShieldGemma and Aegis fail on financial risks. His team built a 12-category taxonomy with about 100 test queries per category. He recommends a guardrail–retrieval–answer–guardrail pipeline, adapting the NIST AI risk framework or MLCommons taxonomies, and red-teaming to measure the risk surface rather than trusting vendor benchmarks.

2025-07-14 · open the original ↗

• This post includes paid content that is not available publicly. See the original source.

2025-07-14 · open the original ↗

• Agents are hidden workflows, so test the steps as well as the answer. Ozdemir says each step can be evaluated: whether the right tool was picked, whether the arguments were right (did it Google the right thing?), and so on. Rather than scoring every micro-action, he suggests a middle level, such as a 100-question dataset that checks only the first tool chosen, ignoring arguments. He calls the first action where agents fail most and says tool selection also suffers from positional bias.
• Measure efficiency and find where extra context stops helping. Ozdemir's second test is cost: tool calls, tokens and time. Ask the same question with more and more context to see whether efficiency improves, where it plateaus, and whether that much context can realistically be supplied in production.
• Every agent must be best at something, or it should be cut. Ozdemir says that in a system of 100 agents, each needs a skill no other agent does better. Otherwise "kill it". He says to test each survivor on its own custom benchmark so the results bubble up into overall system performance.

2025-07-13 · open the original ↗

• Judge multimodal models on the task, not on the modality. Ozdemir says multimodal evaluation is mostly the same as text evaluation: MMMU is MMLU-style multiple choice with images, and for visual question answering the answer can be judged whatever the architecture, whether an omni model such as GPT-4o projecting inputs into text-like tokens or LLaVA. What matters is the goal: trivia about images, or reading an image to use in another task. "Multimodal" itself is ambiguous (audio, video, documents, 2D or 3D images).
• Grading image outputs depends on a judge model that may not be good enough. Ozdemir notes there are hundreds of multimodal benchmarks and dedicated judges such as LLaVA-Critic. Per his description, it scores an image–question–answer triple on a 0–100 rubric or picks the better of two answers and explains why. But when the output is an image, the evaluating model must itself understand images well, and he calls that circular.

2025-07-13 · open the original ↗

• This short is a travel clip and event plug, not commentary. Standing at the world's largest covered wagon in Lincoln, Illinois — 25 ft tall and 40 ft long, bought by the city for $10,000 — he jokes about it and promotes a 1–3 p.m. "meet in the middle" town-hall appearance with another speaker the next day at the Logan County Fairgrounds.

2025-07-12 · open the original ↗

• Perplexity measures a model's confidence, and confidence is not truthfulness. Ozdemir says perplexity, a decades-old metric only recently used as a proxy for hallucination, falls as token confidence rises. It works when candidate answers can be compared: "Mars" for "the red planet" might score 1.3 against higher values for Jupiter or Venus. Without alternatives you need a threshold, and since there is no textbook value for a good perplexity, he says you might as well write a rubric.
• Common tokens get low perplexity whether they are right or not. Ozdemir says perplexity depends on how often a token appeared in training, so the wrong answer "Earth" can also score low because the model has seen it so often, not because it is confident it is right.
• An LLM cannot read its own token probabilities. Ozdemir says sampling from the distribution is done by the hosting system, so any sense that the model doesn't know an answer has to arise in its parameters. He points to probing the model's internals, linked to the idea of world models, as the way to see what it is "thinking".

2025-07-11 · open the original ↗

• Temporal makes failure inconsequential instead of trying to prevent it. Somal, SVP of engineering at Temporal Technologies, says developers code against its SDKs while workers run in their own infrastructure. The server dispatches tasks "like a puppet master" and tells the worker where a failure hit, with retries and backoff set as policy rather than scaffolding. She cites studies putting up to 50% of developer time on error handling.
• A lifted-and-shifted VM gets only coarse recovery. Wrapping a whole old VM in one Temporal activity works but is "a big hammer," she says. A greenfield app wraps each failure-prone step finely so recovery resumes precisely. Steps are finer than old atomic transactions and allow compensation, as in Yum Brands' order processing, where a cancelled taco order must refund the customer and unwind loyalty points.
• Latency cannot be abstracted, so customers choose the region. She says Temporal offers idiomatic SDKs in six languages and runs its data plane on AWS and GCP, with Azure on the roadmap. Customers pick each namespace's region for latency, and Temporal holds a dispatch-latency SLO. Namespaces default to 400 actions per second, which an automatic planner raises, and some customers reach hundreds of thousands.
• Lock-in is to the programming model, not the vendor. Hanselman says hiding a vendor's SDK behind abstractions is self-deception. Somal answers that Temporal is open source and can be self-hosted, and pledges that workflows move between Temporal Cloud and self-hosting with no code changes.
• Durable execution fits AI agents, by her account a lucky coincidence. She says agent terminology maps onto Temporal's existing constructs, and that in a keynote demo at its Replay conference a failing LLM was switched to another vendor live. Hanselman argues that non-deterministic LLMs need retry, fallback and human-in-the-loop recovery. She traces the design to Uber's Cadence project, where even the loyalty programme ran as a workflow.

2025-07-11 · open the original ↗

• The UniFi network ran three years with no drops or failed equipment. At this Idaho lake house, internet arrives at a distant boat house from a wireless ISP (about 50 Mbps across the lake) and a since-installed Starlink Gen 2 dish on a fabricated dock mount, and is carried by VLAN to a Dream Machine SE in the main house; the host kept firmware updated remotely.
• Access points were moved below DFS to stop interfering with the ISP. The wireless ISP reported interference from auto channels; its link runs above the DFS range, around channel 153, so the host manually staggered every access point and Crosstalk's own point-to-multipoint gear onto lower 5 GHz channels, set to 40 MHz width except the busiest main-house access point at 80 MHz.
• The point-to-multipoint link had failed, masked by access-point mesh. One antenna lost power when the client shifted the network rack and half-unplugged it, and a dead 24 V passive PoE injector downed another, replaced from a spare. Mesh kept the back house online, but the host says a point-to-multipoint link is always more reliable than meshing at distance.
• Growing trees have since degraded the back-house wireless link. Two trees now block line of sight to the antenna; the owner, who already planned to remove them, said this gave him the motivation.
• Plasti Dip on the outdoor access points held up through three years. The host contrasts it with a box spray-painted black, which weathered far worse.

2025-07-11 · open the original ↗

• Tech platforms are capturing an ever larger share of advertising. The host reports global ad spending growing about 6% a year, with Google, Meta, Amazon and ByteDance taking more than half the market last year, up from a third five years earlier. Eight of the ten largest ad sellers are now tech firms. He relays Zuckerberg's promise that brands will soon just state a goal and budget to Meta, and Altman's claim that AI will handle 95% of what marketers use agencies for.
• AI strengthens their grip through targeting, measurement and automatic ad creation. Per the host, Meta says Advantage+ raises return on ad spend by 22%, and Google's Performance Max reportedly lifts sales over 10%. He adds that AI search queries run two to three times longer, revealing why people search. A startup using COVID contact-tracing algorithms found Delta's Olympic medal-ceremony sponsorship beat its other promotions. A betting company aired a cinematic TV ad made with Google's AI tools in 48 hours for $2,000.
• Investors are punishing agencies despite their creativity argument. Ad executives argue that models built to produce the most likely output can't make standout campaigns, and some agencies are moving from hourly billing to flat fees. The host reports that four of the five big holding companies' shares have fallen since the start of last year. WPP is seeking a new CEO, Omnicom and Interpublic plan to merge, and only Publicis has avoided the slump, by convincing the market of its AI investment.
• Brands now have to win over chatbots, whose preferences follow their training data. The host reports that Evertune, posing tens of thousands of queries, found models disagree on bed-sheet brands: Casper is popular with most models except Gemini, and Meta's bots favour a different brand. This is attributed to Google and OpenAI licensing Reddit content while Meta leans on its own platforms. Tactics include old-school PR, converting infographics to text, and long product descriptions that only an AI would read.
• The host expects shopping agents to become advertising's next audience. He cites Google's agent that alerts users when prices fall below set levels, and predicts AI will create and target ads that a mix of humans and agents act on. His lesson is that this disruption differs from the shift to digital because it automates creativity itself, and that listeners should ask which parts of their own roles are automatable.

2025-07-10 · open the original ↗

• Benchmark answers can't be kept offline without someone owning them. Ozdemir says a benchmark must ship with answers so results can be validated, which puts them on the internet and invites contamination. A hidden, Kaggle-style answer set, which the host suggests a university such as Berkeley could administer, raises the question of who holds the answers.
• Blind arenas measure preference, not knowledge. Ozdemir likes blind human judging in Chatbot Arena, run by Joey Gonzalez's lab at Berkeley, as a gauge of usability. But casual users bring no structured questions and may pick whichever model talks the way they like. He links this to the sycophancy problem OpenAI attributed to relying on thumbs-up feedback. Aggregation stabilizes the result, but without reference answers it is unclear how correctness is judged.
• Allegations that Llama 4 was tuned for the arena show how unverifiable this is. Ozdemir repeatedly stresses these are only allegations: that Meta submitted to the arena a version of Llama 4 different from the one released. He says such rumours arise when people notice discrepancies, and there is no way to prove them either way.

2025-07-09 · open the original ↗

• Crosstalk offers one RingCentral customer a free migration to its PBX. The company's own offer, framed as an escape from per-extension monthly pricing, covers full project management and number porting onto its open-source-based Crosstalk PBX with no labour charge, for a company with 10 or more extensions and under 6 months left on its contract, in exchange for appearing in a video case study.
• Crosstalk claims switchers save thousands of dollars a year. The host gives no figures beyond that and says customers end up owning their system rather than renting it.

2025-07-09 · open the original ↗

• Mark vibe-coded a better cd but babysat it throughout. His "MCD" is roughly 800–1,000 lines of Rust plus shell integration. It searches the current directory, then up, then down, with tab completion, and is narrower and less fuzzy than Zoxide. He built it in Copilot agent mode and found Sonnet 4 better than GPT-4.1 or o1 at what he calls the limit of frontier capability. He typed no code, but it kept regressing, and the tests and checking added up to a few hours.
• Hanselman built a study-guide site without typing code. He used Claude Code to break a roughly 12-page LaTeX PDF of LLM interview questions into a site with AI answers, definitions, research links and a progress tracker in local storage. He puts it at one and a half to three hours. He says the hardest part was checking that citations were real, and plans to rebuild it in other agents and models to compare them.
• M365 Copilot invented a nonexistent project in detail. Mark says he asked it about a made-up project named in an email. Its chain of thought said it found nothing and should admit that, then it produced pages of dates, milestones and team structure. He frames the question as whether the model is grounded or an improv partner, and says he passed it to the AI red team.
• UI design swings between minimalism and ornament. Mark traces cycles from Windows Phone's flat monochrome back to shadows and colour, and reads Liquid Glass as a fresh look to signal newness. Hanselman questions its value beyond burning GPU, citing reports of phones running hot, and praises the Dynamic Island as a genuine idea. He insists his view is not a Microsoft-ism.
• Vibe coding converges on statistical mediocrity. Mark calls a generic Bootstrap-style site the perfect thing to vibe because the web is full of examples. Hanselman argues that ground-breaking work will need bespoke, unvibed effort. Asked about an AI-optimised programming language, Mark answers that popular languages are already Turing-complete equivalents that differ mainly in preference and ecosystem.

2025-07-09 · open the original ↗

• Trivia benchmarks earn their place as cheap hallucination tests. Ozdemir says static question-and-answer sets such as PersonQA and SimpleQA quickly show how often a model confidently states something untrue. If a model isn't expected to recall facts, those benchmarks matter less, but he says every benchmark generally has its purpose.
• Basic-fact hallucination is still high, so niche trivia adds little. Ozdemir cites OpenAI's own figure that o3 hallucinates as much as 40% of the time on PersonQA and SimpleQA, answering from its own knowledge without web access. Given that, he asks why a benchmark such as Humanity's Last Exam needs StarCraft trivia, when TruthfulQA covers general knowledge such as "what happens if you eat watermelon seeds?" The host notes his own use of o1 and o3 is mostly inside Deep Research, where answers are grounded in web results.

2025-07-09 · open the original ↗

• Keeping pace and staying relevant are the main learning obstacles. Mulcahy, who works on AI adoption in Microsoft's Garage, says significant model, research or tooling updates land every week, so a skill can change before it is learned. She adds that generative AI is non-deterministic: in place of looking answers up in the docs, users must stay the human in the loop and think critically.
• Start with curiosity and one repetitive task of your own. She recommends carving out even 15 minutes each morning to try something, and aiming first at a productivity gain on something done again and again, rather than generic tutorials. Early tools that disappoint get abandoned, she says, but they change fast enough to be worth revisiting. Asked by Giard, she says the same advice applies to non-technologists. Giard adds his own example of learning ASP.NET about 25 years ago by building a site for his favourite sports team.
• You only learn its limits by using it painfully. Her closing advice is to "eat the dog food": use the tools for every question, from photographing something to ask what it is to finding hidden gems on vacation. She proposes committing to a week of non-stop use, then deciding, and says that if you cannot name what is painful in a tool, you are not using it enough.
• AI evolves jobs rather than taking them, with humans directing. Mulcahy says the people who see it this way collaborate with the systems as "creative director", bringing strategy and critical thinking while handing off predictable, formatted work such as weekly reports. Her example is suggesting that her brother, a teacher, build a lesson-plan format to iterate on so he can spend more time on his students.
• Over-reliance and unmet expectations are two separate problems. Over-reliance means skipping critical thinking. She speculates, while saying she does not know what the research shows, that it may erode writing and communication skills, which she calls vital because a good communicator is probably a good prompter. Unmet expectations come from multi-system tasks such as producing a newsletter: interoperability is only starting to be addressed by protocols like MCP. Until then, getting an agent to do it all may mean writing some code.

2025-07-09 · open the original ↗

• This post includes paid content that is not available publicly. See the original source.

2025-07-09 · open the original ↗

• Right-wing audiences have believed a long list of falsehoods. Addressed to viewers angry about the Epstein list, he runs a sarcastic list: Obama born in Kenya, wind turbines causing cancer, sex changes in schools, post-birth abortions, Pizzagate, cats and dogs being eaten, ivermectin curing COVID, the Sharpie hurricane map, Antifa on January 6 — asking why Trump pardoned them — and the FBI planting evidence at Mar-a-Lago, which he says Trump's own attorneys disclaimed.

2025-07-08 · open the original ↗

• The Slate 7's touchscreen is functional, not decorative. It shows WAN and SSID status, offers QR codes for clients to join, live speed and load stats, and toggles for VPN (choosing between client profiles), AdGuard Home and Tor; the host says it has replaced his use of the physical side toggle. GL.iNet's Wi-Fi 7 travel router also has MLO, dual 2.5 Gb WAN/LAN, USB 3 for storage or tethering, and 5/9/12 V input.
• Factory-default setup runs through a short browser wizard. The admin page is at 192.168.8.1: join the GL-BE3600 SSID, scan the bottom QR code or wire in; the wizard sets an admin password and SSIDs, with an optional 160 MHz bandwidth scan that adds time. The host says GL.iNet routers have always got past hotel captive portals for him, though he prefers wired WAN.
• Dynamic DNS is useless behind a hotel's double NAT. The host says remote access there needs GL.iNet's AstroWarp SD-WAN or Tailscale instead. Multi-WAN supports failover or load balancing; he runs Starlink primary with campground Wi-Fi as backup.
• VPN client use is where the router shines. The host uses WireGuard with Private Internet Access, disclosing an affiliate link; AdGuard Home may block VPN providers and need disabling during setup. A home VPN is added by uploading a WireGuard config file as a new client group.
• Network acceleration, on by default, breaks the screen's bandwidth stats. Disabling it under network settings restores them. The host notes GL.iNet had the router on sale for Amazon Prime Day at recording.

2025-07-08 · open the original ↗

• Contamination and closed training data make public benchmark scores unverifiable. Ozdemir says many labs decontaminate training data with n-gram or embedding-similarity checks, which rephrasing defeats. He cites a paper finding that Llama 2 trained on just-rephrased test data would have beaten GPT-4 on common benchmarks. Without released training data, claims rest on trust: per TechCrunch, Meta's VP of generative AI had to deny manipulating Llama 4's scores. OpenAI pulled a sycophantic model within a week in May 2025 and attributed it to a changed RL reward signal.
• Leaderboard tables hide how the scores were obtained. Ozdemir says small print such as "five-shot CoT" means the model only won with few-shot and chain-of-thought prompting. The real takeaway is that the model responds well to those techniques. He puts the onus on frontier labs to say how they got their numbers and on consumers to read them, not on benchmark makers, whom he credits with careful vetting. He questions trivia such as StarCraft and League of Legends questions in Humanity's Last Exam, unless what's tested is a model recognizing its gaps and looking things up.
• Stress-test multiple-choice benchmarks by shuffling answer order and raising temperature. Ozdemir says transformer LLMs have a positional bias toward early options, smaller in recent models. With MMLU-style benchmarks he suggests asking the same question 10–20 times, moving the right answer around and varying temperature, and expecting consistent answers. He notes OpenAI's playground now allows temperature up to 2, which he says produces gibberish.
• Build your own test set and chase your own leaderboard. Ozdemir's first question to clients is "how do you know your AI is working?", and the usual answer is the best benchmark score plus an intern's spot checks. He recommends spending about a week on a domain test set, with human-reviewed synthetic data allowed, which turns experimentation into an internal leaderboard. His example: Stripe's and eBay's domain embedding models would probably score abysmally on MTEB but serve fraud detection and recommendations. He also recommends decontaminating your training data against your test set, for example with an LLM fine-tuned to detect rephrasings.
• LLM judges with rubrics work once validated against human graders. The host describes fine-tuning a 7B Llama with LoRA for natural-language-to-JSON. He found high inter-rater reliability between scarce, disliked human evaluations and an expensive LLM judge, then used the judge alone. Ozdemir says different LLMs score the same rubric differently, so the right judge is the one that matches humans. He adds that such rubrics can serve as rewards for RL algorithms such as GRPO or PPO.

2025-07-08 · open the original ↗

• VFX quality probably peaked 10–15 years ago. Souter, a computer graphics PhD at SKY ENGINE AI, says the technology can now render anything, including fur and fluids that were once hard. He argues the work goes around the world to cut budgets, to artists with less experience and less time to polish. He welcomes YouTube critics as possibly keeping the industry honest.
• Computer vision must see the whole frame, unlike a human viewer. He says VFX steers the eye to the part of the frame telling the story, but a vision model has no such bias and cannot know what to discard, so the lesson inverts. Game engines take shortcuts to look plausible to humans at 120 frames a second. SKY ENGINE AI instead uses physically based ray tracing.
• Synthetic data covers rare, dangerous or impossible-to-capture cases. He cites costly metal defects, drones that would be shot down filming enemy vehicles, and a Japanese firm that filmed its van drivers for about three years to catch drowsiness, with some drivers hurt. Hanselman adds that stacked edge cases, such as a yawn, an overpass shadow and flashing headlights, can be synthesised. Both raise privacy as a reason too.
• Synthetic data must be realistic, and its balance is a design choice. Plastic or uncanny images open a domain gap, Souter says. A platform can set, for example, 20% for each of five ethnicities instead of accepting whoever signs up, and generate more data where a model underperforms, such as on glasses wearers. He says scarce real data should be kept for validation, where results should be close to the synthetic tests.
• Sensors beyond visible light need proprietary renderers. Souter says companies need full access to their renderer's code to simulate near infrared or millimetre-wave radar, which he calls useful for detecting children left in vehicles. He adds that general models such as Segment Anything often miss the very specific things a customer's camera is pointed at.

2025-07-07 · open the original ↗

• The dollar has had its worst half-year in five decades. He reports it lost 11% of its value in the first six months of the Trump administration, the worst performance since 1973.
• Dollar weakness is holding oil far above its fundamentals. Oil rose the day after OPEC announced more than 500,000 barrels a day of additional August production, which he calls a red flag; at $67.80 he thinks it should be well below $60. He expects that if the dollar rebounds even a couple of percent, oil could fall drastically given the OPEC increases.
• Tariffs and policy uncertainty have shrunk demand for dollars. He argues tariffs cut imports, sending fewer dollars abroad, while investors move out of dollar-denominated stocks and bonds over US uncertainty, the tariff policy, the debt-expanding "big beautiful bill" and a high debt-to-GDP ratio.
• A weak dollar argues for raising rates, not cutting them. He says the weak dollar and tariffs keep inflation from falling, so a Powell rate cut now could add inflation and hikes may be needed; he faults the budget bill for adding about $3.5 trillion to the deficit and further weakening the dollar.
• Trump's policies are aiding de-dollarization. He says over half the world is working to remove the dollar as reserve currency and US policy is helping, and argues business success does not qualify Trump to run the economy, which he says Trump plainly does not grasp.