Dr. Jon Krohn

2025-03-09 · open the original ↗

• To eliminate hallucination, don't let the LLM generate answers. Burkov says RAG cuts hallucination sharply, by his rough figures from perhaps 20–30% of answers to 1–2%, but not to zero. For his company's closed-domain labor-market product, the team instead fills predefined templates ("the average salary for a nurse in San Francisco is…") with numbers pulled straight from internal APIs.
• Showing the system's interpretation makes remaining errors visible. The LLM only parses the user's string into structured fields, which are normalized to internal taxonomies (e.g. "JS" to JavaScript) and displayed as labels before any number appears. Errors can still come from interpretation or imperfect scraped data, but Burkov says the chatbot's answer matches what the conventional UI would show; the LLM is used "to understand but not to inform".
• Bolted-on uncertainty warnings can be overcautious. The host reports that Claude now often says a topic is niche and should be double-checked, including on questions he doesn't consider niche, which he takes as a layer on top of the core model that still needs tuning.

2025-03-08 · open the original ↗

• AGI is possible but as speculative as teleportation. Burkov declines to give a timeline, saying only that since evolution produced human minds, something analogous might be built.
• Infinite-horizon planning is what LLMs, and other animals, lack. He contrasts crows that use cars to crack nuts and apes that keep a stick overnight but discard it after three or four days without bananas, with humans who plan for generations, as with climate change. LLMs are reactive: an "agent's" agency comes from its system prompt, not itself, so it can be violated unpredictably. The host adds his own hypothesis that AGI may need brain-like modules, such as a prefrontal-cortex analogue, rather than a scaled-up single architecture.
• LLMs cannot tell what they don't know. Burkov asked OpenAI's o3-mini whether his new book, The Hundred-Page Language Models Book, released after its training cutoff, is good; its visible chain of thought admitted knowing nothing about it, yet the answer confidently said experts and readers praise it. He says a Qwen model fine-tuned to answer "I don't know" about obscure people denied knowing him, then named him as author of his earlier book, showing such fixes are patches, not self-knowledge.

2025-03-07 · open the original ↗

• dbt's real value is documented, agreed business logic. In a clip from episode 861, a former pro athlete turned data engineer says dbt let her team embed business logic agreed with stakeholders, with field definitions (dbt docs, now auto-generated, plus reusable doc blocks) that force agreement on what terms like "active customer" mean and help new engineers. She describes Sigma, sitting on Snowflake, as easing adoption for Excel-minded PMs.
• LLM prompts need a compiler the way HTML needed React. Gupta, CEO of Boundary (episode 859), says BAML ("basic-ass machine learning") was inspired by React catching an intern's missing closing div that could break a news feed. Because every codebase will hold more prompts each year, he argues developers consuming models need type-safe tooling with instant error feedback, as TypeScript gave JavaScript.
• A tabular foundation model is spreading across scientific applications. Hutter (episode 863) and the host note v1 was already used in a Science paper on chemical proteomics, and an "awesome TabPFN" repo Hutter had just created lists about a dozen applications, including insurance, fault classification, finance, wildfire propagation and algal blooms. Version 2 handles more data types, missing data, outliers and larger datasets, and can be fine-tuned for domains its authors know nothing about.
• A mandatory paid discovery phase made Pandata scalable. Al-Dhubaib (episode 865) says returning clients valued plain speaking and a 30-day to six-week "discovery and design" engagement, so he made it compulsory, priced then at 12,000 and now 50,000. The pipeline briefly dried up, one of three times Pandata's bank account fell below a month's expenses, but he then landed three of his biggest clients, two of whom stayed about six years until the exit; his lesson is the art of saying no.

2025-03-06 · open the original ↗

• Executives' agent hype reflects their shareholders' interests. Against the host's list of boosters, Burkov says Nvidia's CEO talks up agents because it implies buying more GPUs, and that Salesforce once declared traditional software dead in favour of SaaS, which did not happen, and now says the same of SaaS. He distinguishes Andrej Karpathy as careful to describe possibilities rather than predict them.
• A single agent is a rebranded LLM, useful only in-distribution. Burkov's best case for agents is gathering and summarizing web information, because web data is closest to what LLMs were trained on; he questions how big that market is, likening some of it to a cron job running Google searches. Dropped into a company's intranet of legacy systems they have never seen, he expects agents to "break their teeth" quickly.
• LLM-based multi-agent systems are effectively undebuggable. Burkov, whose PhD was in multi-agent systems, explains that distributed systems are already hard to debug because you cannot pause all asynchronous actors at once, and LLM agents add black-box internals you cannot step through. Reliable multi-agent systems need full control of each actor and inspectable communication protocols.
• Agents need human-level trustworthiness before being combined. Burkov argues that even human systems guard against unreliability with redundancy, such as two keys turned far apart, and that today's agents are nowhere near as reliable as a person, so multi-agent systems built on them are "a recipe for disaster". He says no one has a clear idea how to reach a trustworthy AGI.

2025-03-05 · open the original ↗

• TalentNeuron turns scraped job postings into workforce insights with NLP. The host cites Burkov as having previously said the company makes 95 million daily predictions from 700 million job-posting data points. Burkov describes some 35,000–40,000 rule-based robots downloading postings, then models that extract and normalize skills, telling "excel in Word" from the skill Excel, and predicting industries with BERT and RoBERTa since around 2018–2019.
• Multilingual models removed the need to label data per language. Where the team once built separate salary extractors for about 25 languages, Burkov says they now reuse labeled data from other languages to cover new countries, a capability he says he would have called impossible ten years ago.
• LLMs now build about half of the company's scrapers automatically. Given a corporate home URL, the system finds the careers section and writes extraction rules for titles, locations, dates and descriptions; the remaining harder half is still done by engineers.
• Think hard before putting an LLM in front of users. Burkov tells decision-makers to think "30 times", warning that LLMs work in-distribution but businesses cannot know what is in-distribution because training data is hidden, even for open-weight models. Systems that pass developers' tests can fail arbitrarily on real production data, with no way to detect the failure or know whether more fine-tuning examples will fix it, costing reputation and angry customers.

2025-03-04 · open the original ↗

• LLMs are for prototyping; production still needs trained, controlled models. Burkov, ML lead at TalentNeuron, warns that businesses cannot know whether their problem is in-distribution because training data is hidden, so systems that pass tests fail unpredictably on real users. His four-person team uses instructed LLMs as placeholders at the many decision points in a pipeline, then replaces the weakest links with fine-tuned or purpose-built classifiers, because errors such as 15% at several steps compound toward zero end-to-end success.
• Agents won't work in real life, and multi-agent systems can't be debugged. Burkov, whose PhD was in multi-agent systems, calls a single agent a rebranded LLM with the same out-of-distribution problems, useful mainly for web information gathering. Asynchronous black-box agents cannot be paused and stepped through, so he calls building multi-agent systems on them "a recipe for disaster" until agents are as trustworthy as humans. He attributes CEO hype from Nvidia and Salesforce to shareholder interest.
• LLMs lack infinite-horizon planning and a sense of what they don't know. Burkov likens AGI's prospects to teleportation and declines a timeline. Humans plan for generations where apes discard a tool after three or four days, while LLM agency comes only from the system prompt. His test of OpenAI's o3-mini on his new book, released after its cutoff, produced a reasoning trace admitting ignorance but a confident answer saying experts praise it.
• Closed-domain chatbots avoid hallucination by keeping LLMs from writing answers. At TalentNeuron the LLM only parses queries into structured fields, which are normalized and shown to the user, while numbers come from internal APIs into fixed templates. Burkov contrasts this with RAG, which by his rough figures cuts hallucination from perhaps 20–30% to 1–2% but not to zero.
• DeepSeek removed the frontier labs' moat. Burkov says it trained a state-of-the-art model for 5 to 10 million instead of 20 to 50 million, published a recipe others have reproduced, and charges $2 per million output tokens against o1's $60. Its R1 training used automatically verified rewards (code compiles, math matches) rather than expensive expert-written data. He adds that open-weight models are not open source without their training data.

2025-03-03 · open the original ↗

• Sell to the buyer where your business creates the most value. Al-Dhubaib says firms had long approached Pandata because buying a data science team is easier than building one. When the head of Further (then Search Discovery), a fellow Case Western alumnus seeking to move into cloud, AI and advanced ML, offered to buy within five minutes of meeting, he first asked about core values to judge whether his clients and team would thrive there.
• Upfront compatibility enabled an unusually fast deal. Al-Dhubaib found the two firms shared problems, attitudes and pace despite differently named values; diligence and paperwork ran from January to March of last year, about three months. He attributes the horror stories he hears about other exits to skipping that matchmaking, partly because it is hard to say no to good-sounding offers.

2025-03-02 · open the original ↗

• Evaluation design and LLM operations are today's scarce data science skills. Al-Dhubaib says what "data scientist" means has changed drastically over ten years; he now seeks people who can design statistically sound evaluation and quality control for unstructured, multimodal data without leaking what they predict into the data, and ML engineers experienced in deploying language models and diagnosing drift or odd behaviour, which he calls AI Ops.
• The evergreen traits are soft skills, not technical ones. These are explaining plainly, a drive to learn, and comfort with ill-formed problems. Whatever the technical skills of the moment, Al-Dhubaib says these core values from Pandata are what he consistently looks for and recommends data scientists cultivate to stand out.

2025-03-01 · open the original ↗

• Without an upfront value hypothesis, AI projects fail on ROI. Al-Dhubaib says data scientists get enamored with model accuracy instead of asking which decisions the model influences, what a successful intervention is worth, and how often the decision will actually change.
• A risk score is useless unless it tells providers what to do. In a hospital project on solid-tumor cancer patients, whose national readmission rate he puts at about 25%, clinicians responded to risk alerts with "now what?". The team added interpretability showing why a patient was at risk, such as medication, comorbidity or a zip code with unreliable transport, pointing to interventions like social assistance.
• Stress-test the value estimate with the client before investing. In the readmission example the team assumed it could identify 80% of at-risk patients but only 20% would receive an intervention, valued the project on that basis, and asked whether it was still worth the investment, revisiting the hypothesis as models were validated.

2025-02-28 · open the original ↗

• Investors are pricing de-extinction on its side technologies, not revenue. The host reports that Colossal Biosciences raised a $200 million Series C at a valuation over $10 billion, nearly sevenfold its valuation a couple of years earlier, with no revenue yet and its main projects not projected to complete until 2028.
• The method is genome comparison, CRISPR editing and a surrogate. As the host describes it, Colossal maps the extinct species' genome, compares it with a close living relative (the Asian elephant for the mammoth), edits the relative's cells with CRISPR to match, and implants the resulting embryo in a surrogate; it has also built artificial-womb technology the host says could apply to human fertility treatment.
• Colossal makes money three ways while chasing its core mission. The host lists spin-offs (one using microorganisms to break plastic waste into organic matter, and Form Bio, a computational biology platform for gene therapies, both with millions in further funding), conservation work with governments on species such as the northern white rhinoceros, of which two females remain, and an emerging biodiversity-credits market modelled on carbon credits.
• De-extinction needs good DNA, a living surrogate and a surviving habitat. On those conditions the host rules out dinosaurs; he names the dodo and Tasmanian tiger as active Colossal projects, the passenger pigeon as a Revive & Restore project, and the Caribbean monk seal, quagga, great auk and heath hen as other candidates.
• The host's lesson is that impossible goals throw off valuable byproducts. His opinion is that each of Colossal's indirect applications could eventually generate billions of dollars in revenue on its own, and he urges listeners to use AI tools such as OpenAI's and Google's Deep Research features, new "as of the past few weeks", to dream up ambitious goals.

2025-02-27 · open the original ↗

• Repeat clients, not interest, revealed Pandata's product-market fit. Al-Dhubaib says that because he bootstrapped, paying customers were his only source of growth; when he asked one of the few returning clients why it came back, the answers were his firm's plain, jargon-free approach and an upfront scoping phase it had stumbled into.
• A mandatory paid discovery phase screens out the wrong clients. He made a 30-day to six-week "discovery and design" engagement compulsory, priced at $12,000 at the time and $50,000 now, to find "where the skeletons are" and the unknown unknowns; it filtered out clients who weren't serious and those who didn't accept the data science process and would end up disappointed.
• The filter nearly broke the firm before it paid off. Al-Dhubaib says his pipeline dried up and the bank account fell below a month of expenses, one of three such times, but in that same period he landed three of his biggest clients, two of which stayed for about six years until Pandata's exit.
• Saying no to small, low-value work is what lets a consultancy scale. He calls ruling clients out a counterintuitive lesson that frees time for higher-value work, and says the acquirer's due diligence flagged about three points in the financials that matched the moments he had regretted too.

2025-02-26 · open the original ↗

• The failed startup's free research was the real product. Al-Dhubaib says his first venture, built out of computational neuroscience work at Case Western, raised money, entered an accelerator and collapsed in about a year and a half; hospital researchers he had been doing free pilot research for then told him they had depended on it, and would pay for it.
• Pandata was founded on a regional data science talent gap. At its start, at the end of 2015 and start of 2016, he says Northeast Ohio had fewer than 150 data scientists, about a third of them at IBM, the Cleveland Clinic or Progressive Insurance, while other large local enterprises had one or two.
• The firm survived by moving to whatever was still hard. He says it shifted from analytics and business intelligence to machine learning, then to machine learning in heavily regulated environments as each layer became commoditized; after about eight bootstrapped years it was acquired by Further, where he is now head of AI and data science.
• "Make it boring" became a core value because clients who understand stay. Al-Dhubaib says many data scientists of the time liked being opaque, while he found clients who grasped the work hired the firm and stayed longer, so he explains complex mathematics with cats, dogs and memes.

2025-02-25 · open the original ↗

• A mandatory paid discovery phase built Pandata's durable client base. Al-Dhubaib made a 30-day to six-week "discovery and design" engagement compulsory, at $12,000 then and 50,000 now, to weed out unserious clients and those who would not accept the data science process; the pipeline dried up and cash fell below a month of expenses, then he landed three of his biggest clients, two of whom stayed about six years until the exit.
• Mid-sized regulated work too small for the big firms was the niche. He says Pandata served hospitals, life sciences, higher education, energy and utilities and some defense work, built insurance and security processes "way more secure than any of our clients", charged a premium, occasionally took projects from McKinsey or Accenture, and that this expertise was a big reason Further acquired it.
• Building users' trust lifted a failed model without changing it. In his telling, an ensemble model for a health-tech client with about 50 million patient lives found 30% more people eligible for government assistance, but its pilot converted 2 in 100 mailings against a baseline of about 10, because staff with bonuses tied to selection distrusted its unfamiliar picks; after training sessions and per-prediction explainability, the same model grew reach by 18%.
• Projects need a value hypothesis about decisions, not model accuracy. For cancer readmissions, with a national rate of about 25%, Al-Dhubaib says providers asked "now what?" until the model explained the drivers of risk, such as poor transport access; he now stress-tests value up front, for example identifying 80% of at-risk patients with a realistic 20% intervention rate.
• An acquisition works when the buyer's values and clients fit. He says he asked Further's CEO about core values within minutes of being offered a deal, to judge whether clients and staff would carry over, and diligence closed in about three months between January and March of the previous year; his hiring values, he says, are approachability, winning together, cultivating trust, pursuing growth and taming uncertainty.

2025-02-24 · open the original ↗

• A university group could not build AutoML tools for the masses. Hutter says that after 10 to 12 years co-leading the AutoML community, whose best-known tool, Auto-sklearn, wraps scikit-learn to choose methods, preprocessing and hyperparameters, a lab with a few research engineers only ever built tools fit for researchers; reaching a broad user base, especially with TabPFN, needs a commercial entity with strong engineers.
• Prior Labs builds products while the university keeps the moonshots. Hutter, co-founder and CEO of Prior Labs, says he will keep an academic co-affiliation and research tabular-data questions there, such as how the learned algorithm works, how it shifts with the priors and what its failure modes are; PhD students gain engineers who can turn their papers into products.
• Prior Labs is hiring broadly and putting community before sales. At recording, Hutter says a funding announcement was due two days later; the company was hiring AI scientists, ML engineers, backend engineers and community staff, with sales left for later.

2025-02-23 · open the original ↗

• Even TabPFN version 1 found real scientific and industrial uses. The host cites a Science paper on large-scale chemoproteomics for ligand discovery that used TabPFN in its inference, and an "awesome TabPFN" GitHub repo listing about a dozen applications across health insurance, factory fault classification, finance, wildfire propagation and biology.
• A generic method lets non-experts fine-tune into specialist domains. Hutter says he knows nothing about chemical proteomics but could fine-tune TabPFN on that field's data to beat the general model, and he names algal-bloom prediction and many medical applications; he expects finance uses exist but go unpublished.
• Prior Labs wants the open-source community to carry adoption. Hutter says he created the applications repo the day of recording and invites pull requests; within a couple of days of the Nature paper, an open-source interpretability repo had integrated TabPFN, and the host notes version 2 handles missing data, outliers and larger datasets.

2025-02-22 · open the original ↗

• TabPFN's prior is a generator of random structural causal models. Hutter credits first author Noah Hollmann with a method that samples graphs in which features and the target are nodes with unknown causal direction; real data never reveals the true graph, so the ideal prediction is a Bayesian posterior that weights each candidate model by how well it fits.
• The network learns to approximate that posterior in one forward pass. Hutter says TabPFN never stores the roughly 130 million structural causal models used to train it; it sees only the raw data and has learned to interpolate across all of them, which the host frames as a substitute for the trillions of tokens language models get and tabular data lacks.
• Training only on synthetic data rules out test-set leakage. Hutter says reviewers rejected the first TabPFN v1 submission because its performance looked too good, suspecting tuning on the test set; the team dropped a component that made gradient updates on real datasets, lost almost nothing, and since then never touches real data in training.

2025-02-21 · open the original ↗

• o3-mini high is the top public model on key reasoning benchmarks. The host says that in its high mode, the one with the most inference-time compute of three (low, medium, high), OpenAI's newly released o3-mini beats o1-mini, DeepSeek R1 and the full-size o1 on the AIME math, Codeforces coding and SWE-bench Verified benchmarks.
• o3-mini matches or beats o1 at a fraction of its price. The host puts o1 at $60 per million output tokens and o3-mini at 7% of o1's price on both input and output; he says o3-mini costs about twice as much as R1 on DeepSeek's own cloud in China, but about half as much as R1 run on a US cloud provider.
• The host rates the release below R1 because o3-mini is proprietary. He says R1's open-source release gives more freedom to adapt the model and to run it on any infrastructure; o3-mini is available to free ChatGPT users through the "reason" button, to paid plans as o3-mini high, and through the API.
• OpenAI's own figures put the full o3 far ahead. The host relays OpenAI's not yet independently verified scores: on AIME, 96.7 against 87.3 for o3-mini high and 79.8 for R1; on Codeforces, an Elo over 2,700 against about 2,100 and 2,000; on SWE-bench Verified, almost 72 against about 49; on GPQA, almost 88 against 80. He expects o3's public release soon and takes OpenAI's past figures as fairly reliable.

2025-02-20 · open the original ↗

• Prior-data fitted networks are trained to compute the Bayesian posterior predictive. Hutter confirms that this is the optimization objective of every PFN, so a Transformer is carrying out Bayesian inference.
• PFNs replace hairy MCMC and variational inference with plain supervised learning. Hutter says closed-form posteriors exist only for simple cases such as Bayesian linear regression or Gaussian processes with known hyperparameters; beyond that, MCMC is slow to converge and variational inference carries approximation error, whereas a PFN samples millions of datasets from the prior and learns with cross-entropy loss to predict held-out points, approximating Bayesian inference "to an arbitrarily strong degree".
• A PFN can integrate over architectures, not just weights. Hutter says standard Bayesian neural networks place a posterior over the weights of one fixed architecture; with a PFN you can place a distribution over architectures too and get the posterior predictive in a forward pass, which he calls Bayesian neural architecture search.
• The limitation is that PFNs give no posterior over weights. Hutter says they go straight to the predictive distribution of y given x, so they cannot say which architecture or weights are right, only give the Bayesian average over all of them.

2025-02-19 · open the original ↗

• Deep learning hasn't helped tabular data because its features already exist. Hutter says tabular datasets are mostly small and unlike one another (omics blood values for Alzheimer's against transactions for fraud), lack the spatial regularity that convolutional networks exploit, and come with human-designed features; a data scientist's higher-level engineering, such as computing BMI from height and weight, is what adds value.
• TabPFN first beats XGBoost and random forests on their own information. He says the first step was an apples-to-apples test using only raw values and labels, not column headers; combining it with language models that read headers is in progress with "some initial results", and he expects it to be "so much better".
• TabPFN does tabular learning by in-context learning, like GPT. Hutter says the whole training set and test inputs go into the prompt and the model outputs the test labels, trained with cross-entropy, so each dataset is one training example; lacking trillions of quality real tables, the team generated all training data synthetically, which he believes makes theirs the first paper to reach state of the art on synthetic data alone.
• Sampling enough datasets from a prior makes a Transformer exactly Bayesian. He cites the team's 2022 paper "Transformers Can Do Bayesian Inference": a model fitted to many datasets drawn from a prior approximates the posterior predictive under that prior, and with an arbitrarily large Transformer and minimal loss it recovers exact results, such as true Bayesian linear regression or a Gaussian-process posterior.

2025-02-18 · open the original ↗

• TabPFN beats boosted trees by learning tabular prediction in context. Hutter says it feeds the whole training set and test inputs into a Transformer prompt and outputs test labels, using the same raw values XGBoost and random forests get and no column headers; adding language-model reading of headers is in progress with "some initial results".
• Training on synthetic datasets alone makes the Transformer approximately Bayesian. Hutter says the model learns from about 130 million generated structural causal models, the prior devised by first author Noah Hollmann, and approximates the Bayesian posterior over them in one forward pass; he believes it is the first state-of-the-art model trained only on synthetic data, and says it rules out test-set leakage, after reviewers rejected the first v1 submission as too good to be true.
• Version 2 turns a proof of concept into a general tool, for small data. Hutter says v1 handled only numerical classification on about 1,000 points and 100 features, and saw around 15 published uses; v2, published in Nature in January, handles regression, categorical, missing, outlier and imbalanced data, with text through the API only, and was evaluated up to 10,000 points and 500 features. He is confident of scaling to 100,000 or a million with combined approaches.
• Untrained on time series, TabPFN still leads time-series forecasting. Hutter says casting timestamps as features (time of day, day of week, day of month and sine and cosine encodings) lets TabPFN forecast any horizon in one pass; at recording it ranks first on the GIFT-Eval benchmark, ahead of Amazon's Chronos and other time-series foundation models.
• Hutter launched Prior Labs because academia can't ship for the masses. He says Auto-sklearn showed that a lab with a few research engineers builds only for researchers; Prior Labs, where he is co-founder and CEO, will build products while his academic co-affiliation pursues interpretability, failure modes and moonshots. The host says the TabPFN library had been downloaded almost a million times at recording.

2025-02-17 · open the original ↗

• Fotsch treats exercise as a productivity tool for desk careers. She says a fixed 30- or 45-minute workout block improves discipline, time management, energy, sleep and confidence at work, and that tech culture undervalues this as a help to being a better data engineer or analyst.
• Small daily movement adds up more than people expect. She recalls logging about 800 steps one day in her first data analyst job despite having worked out, and recommends stairs, walking Zoom calls and walking one-on-ones; someone managing one workout a week should aim for two plus more walks, not jump to six.
• Over-trainers should save all-out efforts for a few sessions. Responding to the host, who says copying two-hour, five-to-seven-day CrossFit regimens leaves him too tired to work, she suggests one or two all-out workouts a week. Otherwise, she says, scale intensity by how you feel and by readiness and sleep scores (taken with a grain of salt), progress gradually, and choose a programme that fits your own schedule rather than someone's on Instagram.
• Consistency, not perfection, is what produces fitness. Fotsch says fitness comes from many days of good choices, made about 80% of the time, rather than one hard week, and that goals can be as simple as feeling good and enjoying training; she saves her hardest efforts for Saturdays so weekday work doesn't suffer.

2025-02-16 · open the original ↗

• After retiring, Fotsch had to stop training like a Games competitor. She says that for months as a full-time analyst she still trained as if heading back to the CrossFit Games, with her identity tied to being strong at elite movements, until she asked whether muscle-ups and handstand push-ups served her current goals and began training for feeling fit "20 30 years from now".
• Short, minimal-equipment workouts got results without hours in the gym. Fotsch says she found she did not need her old intensity to reach her new goals, and that the dumbbell, bodyweight and hotel-room workouts she posted resonated with followers.
• PRVN GO drops competition skills for busy professionals and parents. Fotsch says PRVN's CEO, Nick Johnson, approached her soon after she joined CHG; unlike PRVN's competition-oriented CrossFit programming, the company of champion Tia-Clair Toomey and her husband Shane, PRVN GO omits high-skill movements in favour of conditioning, bodybuilding and core work. The host says the episode is not sponsored.
• A remote community supplies the accountability a class would. She says users who can't get to a class log the same workout and talk with others doing it elsewhere, which counters the excuses that build up after a hard week.

2025-02-15 · open the original ↗

• dbt turns raw warehouse data into stakeholder-ready models, mostly in SQL. Fotsch describes her work at Podium as adding fields and business logic to existing dbt models, later building models from scratch, using dbt's functions to consolidate repetitive code ("SQL on steroids"), and exposing the results in Tableau or Sigma.
• Documented definitions force agreement on what the business logic means. She says dbt docs make every field carry a definition agreed with stakeholders, which matters because something like "created date" or "active customer" may not mean what people assume, and which helps new engineers learn the data's meaning and granularity.
• Generated documentation and doc blocks remove the tedium. Fotsch says dbt now auto-drafts field definitions, which the host calls a good generative-AI use that removes the blank-page problem though output should be checked; doc blocks let a field defined in a staging model be reused throughout downstream lineage. The host contrasts the generic AI fitness tips he found useless on his Whoop.
• Sigma's spreadsheet feel won adoption from product managers. She says Sigma sits directly on Snowflake, removing an ingestion step, and its Excel-like look served both analysts and Excel-familiar PMs, which helps get everyone onto the same numbers.

2025-02-14 · open the original ↗

• Plan with compounding percentages and feedback loops, never linear increments. In a clip from episode 855, futurist Azeem Azhar says nothing grows linearly except birthdays: growth follows S-curves with an exponential phase. He cites the Intel–Windows upgrade loop that drove chip progress, builds dynamic percentages and feedback loops into his own forecasts, and says a linear forecast of S3 storage spend makes "incredibly extreme assumptions", even if the realistic one alarms the CFO.
• Hybrid quantum computing targets scheduling and logistics that classical heuristics only approximate. In episode 851, Florian Neukart describes a Volkswagen production-testing problem with 250-odd test classes, each with subtests, to fit to available staff skills; he also describes fleet routing with delivery windows and empty miles. He concedes quantum results are not provably optimal either, so the classical and quantum answers are compared and the better one used.
• Choosing a foundation model means weighing a long criteria checklist. In episode 853, Kirill Eremenko lists twelve: cost, modality, customization, inference options, latency, architecture, performance benchmarks, language, size, scalability, compliance and licensing, and environmental impact; Hadelin de Ponteves says in practice his choices, such as an Amazon Titan model and a Llama model, came down to which matched his fine-tuning dataset's format.
• Layered, reference-free metrics evaluate agents better than single accuracy scores. In episode 857, Coval's Brooke Hopkins says getting the most-used case wrong matters more than a 99% average, so metrics are combined: a short conversation is bad only if the goal also failed. Coval checks JSON-described workflow steps and function-call arguments, inferring correctness from context rather than golden datasets, and flags agents that take longer than humans or repeat themselves.

2025-02-13 · open the original ↗

• Analytics engineering is the data-modeling layer between ingestion and analysis. Fotsch says that at Podium, data engineering pulled data in and maintained pipelines, analytics engineering modelled the raw data in dbt for stakeholder use, and analysts built Tableau visualizations, working with the modellers on the questions being answered.
• Whether the role exists depends on the company. She says neither CHG, her current employer, nor Homie had a separate analytics engineering team, though the work still happens; at Podium, analysts gradually took on analytics engineering work as the team grew, which she credits with leveling up her technical skills.
• Knowing what tools exist matters more than having used them all. Fotsch says learning from what fails, and knowing where to look, is the core of the job; on her first day she wanted to take ten courses at once, and her sister-in-law, a data analyst, told her it was a marathon, not a sprint. Context on the wider pipeline, she says, helps with stakeholders and with knowing whom to ask.

2025-02-12 · open the original ↗

• Formal training builds the confidence to explain your value in interviews. Fotsch says a degree such as her Arizona State programme, plus Coursera courses she still uses, helps both on the job and in interviews; for analyst roles she prioritized SQL above all, some Python, and the popular BI tools Tableau and Power BI, whose skills transfer.
• A referral from someone who knows you conveys what a resume can't. She got her Homie interview through a friend from the fitness world working on the product side, who connected her to the data manager and could vouch for her work ethic and teamwork; she advises listing target companies and industries and finding anyone you know there, not only people in data.
• In-person meetups build a community worth having even when employed. Fotsch says Utah has many data meetups and the contacts she makes there are people she can later ask how they solved a problem; the host adds that newcomers with little knowledge are welcome at data analyst, data science and AI meetups listed on Meetup.com.

2025-02-11 · open the original ↗

• Fotsch reached the CrossFit Games after years of near misses. The host reads her record: 491st worldwide in the 2014 Open, 113th in 2015 (when she won workout 15.5 outright), 80th in 2016 but seventh at Regionals with five qualifying, fourth at the 2017 Games on a team after an 80% rotator-cuff tear, seventh again at 2018 Regionals, then 33rd in the 2019 Open and 27th at the Games; before that she won two NCAA titles with Cal and twice broke an American relay record.
• A flawed Games appearance pushed her to plan life after sport. Fotsch says the 2019 Games' expanded field of national champions, upwards of 100 per division against the usual 40, was not the Games athletes felt they deserved, and taught her she was not entitled to any outcome; elite athletes, she argues, avoid thinking about what comes next.
• She was already a data nerd from tracking her own training. Fotsch says weighing food, timing meals and sleep, and overlaying wearable data on workouts so her coach could balance load against recovery drew her to data, after her sister-in-law, a Netflix analyst, described the field; her kinesiology master's, she says, overturned a belief that she wasn't smart enough for technical work.
• An ACL tear ended her bobsled bid and hastened her data career. She made the US national bobsled team as a brakeman in 2021, aiming for 2022, but tore her ACL weightlifting months before qualification, and began job applications from bed after surgery while finishing an Arizona State business data analytics master's; a friend's referral got her an analyst role at Homie.
• Her path ran from analyst to analytics engineer to data platform manager. At Podium she moved from Tableau analysis into dbt modelling; at CHG Healthcare, a medical staffing firm, she is senior technical manager of marketing technologies within the data platform team, centralizing scattered marketing data, delegating more and building less. She credits CrossFit's "figure it out" habit for her comfort learning on the job.

2025-02-10 · open the original ↗

• Like a compiler, BAML handles the prompts nobody will hand-tune. Gupta says a decade of performance engineering taught him that compilers beat him on time per dollar, so a team should hand-optimize only its one or two most critical prompts and let tooling handle the other 90%.
• Schema-aligned parsing tolerates the model's malformed output instead of forcing strict JSON. Gupta says his team spent about eight months on an algorithm that coerces a response into the expected data model, fixing missing quotes and commas and strings given where numbers were expected, in under five milliseconds; he says customers save 20% to 30% of output tokens.
• The parser brings function calling to models released without it. Gupta says it needs no model modification, so BAML users were tool-calling with DeepSeek R1 and OpenAI's o1 models at recording, and a customer building a Cursor-like tool for Xcode had the LLM emit code in triple quotes, avoiding JSON escaping, which BAML turned into properly escaped strings.
• Semantic streaming lets chat products show a generative UI, not just text. He describes type-safe partial data driving interfaces such as recipe cards with a progress spinner per section, or live tickers and graphs for stock analysis, and argues a chat startup must ask why its users wouldn't just go to ChatGPT, with better UX as the answer.

2025-02-09 · open the original ↗

• Good RAG depends on data-specific retrieval no general tool can supply. Gupta says his company started on RAG pipelines as one of its 13 pivots but moved away because pulling the relevant documents depends on each dataset, so there is no general RAG solution, just as there is no universal web accordion component.
• JSON pasted into prompts is filler the model must filter out. He calls them the prompt's "likes and ums": quotation marks, escape characters and colons make context parseable for code but harder for the model to read, and that the placement, orientation, size and surrounding text of images also matter; BAML renders the actual prompt live in VS Code so developers can spot what looks ugly, on the reasoning that models are trained on human data.
• Treating prompts as typed functions curbs the contradictions of long English instructions. Gupta says every BAML prompt is a function with type-safe parameters and a declared return type, such as an enum of categories, with each category's rule attached to that category rather than one growing essay; this, he argues, yields shorter prompts written like code.

2025-02-08 · open the original ↗

• Gupta predicts prompt engineering will mature as front-end development did. Responding to a host premise, from his researcher, that it is a "joke discipline", Gupta says he once objected to the term "AI engineer" being used for prompt writers, but now sees prompting as being where web development was with plain HTML, no CSS and no interactivity, before it became a complex craft; he expects it to become a well-studied discipline over time.
• What prompting needs is reliability in transforming data. Gupta frames a prompt as a way to turn supplied data into the data you want, such as classifying a transcript as a doctor–patient conversation or general wellbeing, and says using a model involves more nuance than people credit.

2025-02-07 · open the original ↗

• DeepSeek R1 matches the top models on human-rated leaderboards. The host reports that at the time of writing R1 was statistically tied for first on the overall LM Arena leaderboard, within a 95% confidence interval, alongside GPT-4o and Google's Gemini 2.0 Flash, and that its release coincided with Nvidia's share price falling 17% and the Nasdaq several percent in a day.
• R1's real shock is its tiny training cost. In the host's rough approximation, a DeepSeek V3 or R1 training run costs on the order of millions of dollars against hundreds of millions for o1, Gemini or Claude 3.5 Sonnet; he credits a conceptual breakthrough combining mixture-of-experts with new efficiencies such as DualPipe, a GPU communications scheduler, across a couple of thousand GPUs, about 1% of what Meta and xAI say they buy in a year.
• The host argues US chip sanctions on China have proven ineffective. He says a Chinese firm approached US capability with about 1% of the chips, at about 1% of the cost, using less capable Nvidia chips, and that DeepSeek released V3 and R1 source code and weights under the MIT license, more open than Meta's Llama, which shares weights but not code and restricts companies with over 700 million active users.
• Avoid DeepSeek's app, but run its models privately. The host warns that, per its privacy policy, anything entered in DeepSeek's iOS app, number one on the App Store at recording, is stored on servers in China; he suggests Ollama for private use.
• Efficient training may strand the giant capital raises, the host argues. In his view, recent $6 billion raises by OpenAI, xAI and Anthropic and the $500 billion Stargate project probably assumed models would keep growing by orders of magnitude; he welcomes cheaper training as better for the environment and for building applications.

2025-02-06 · open the original ↗

• Y Combinator partners doubted the Slack-competitor idea that got Gupta admitted. Gupta says he and his co-founder, UT Austin graduates of 2015, tried edtech and the creator economy before applying to Y Combinator with a warmer alternative to Slack; on his fourth application, and third interview, they got in, though the partners told them the idea would not work.
• Fielding batchmates' machine learning questions revealed the real market. He says they joined the Winter 2023 batch as ChatGPT came out, and as more seasoned engineers, one with nine years in distributed systems and the other nine or ten in algorithms and assembly, found themselves answering mostly ML questions; they also concluded they shouldn't be building UIs, and the company went through 13 pivots in all.
• An SDK can't express what's needed; a new cross-language syntax can. Gupta says Python or TypeScript SDKs such as Vercel's AI SDK and Pydantic AI are limited by their host language, and since every software shop, whether C++, Java or Go, will use AI, the answer is a new syntax like JSON, YAML or TOML before it, which became BAML.

2025-02-05 · open the original ↗

• BAML aims to do for prompts what React did for HTML strings. Gupta, CEO and co-founder of Boundary, says that as a Meta intern he saw how concatenated HTML strings let one unclosed div break the news feed, and that React's new syntax combining HTML and JavaScript gave instant error squiggles, state management and hot reloading; BAML, "basically a made-up language", applies that idea to LLM calls.
• Ever more prompts in codebases demand tooling that catches errors. Gupta predicts each year's codebase will hold more prompts than the last, and that models of all kinds will get easier for non-experts to use, which calls for developer tooling for consumers of models rather than their producers.
• Prompts need TypeScript-style type safety and static analysis. He argues that however good models get, their use is still code that must fit a codebase, and that BAML's algorithmic work reframes prompting the way TypeScript's type safety improved JavaScript.

2025-02-04 · open the original ↗

• Prompts need their own language, as HTML strings needed React. Gupta, CEO and co-founder of Boundary, says codebases will hold more prompts every year and that SDKs in Python or TypeScript are limited by their host language; BAML is a new cross-language syntax, like JSON, YAML or TOML before it, that treats each prompt as a typed function with typed parameters and a declared return type, such as an enum whose categories carry their own descriptions.
• A hot-reload loop multiplies how many prompts a developer can test. Gupta says that without BAML he sees people try about five prompts in 20 minutes, while BAML's in-editor tests, modelled on Rust's run-test buttons, take about 5 seconds each, allowing 240 in 20 minutes; it also renders the actual prompt live so developers can spot formatting noise such as JSON blobs and badly placed images.
• Schema-aligned parsing repairs malformed model output and saves tokens. Gupta says about eight months of work produced an algorithm that coerces responses into the expected schema in under five milliseconds, which he says saves customers 20% to 30% of output tokens and gives function calling to models released without it, such as DeepSeek R1 and OpenAI's o1, at recording.
• Next, BAML adds conditionals and versioned data schemas. Gupta says users want control flow, for example routing free users to GPT-4o mini and paid users to o1 while reproducing each in the test loop; a protobuf-like versioned schema layer is meant to stop rigid data pipelines from breaking when an enum grows from five categories to 50.
• Boundary hires on references, not technical interviews. Gupta says the company has never posted a job: candidates email "why I'm awesome" with three achievements, then have a short call, after which Boundary interviews three references in depth on the technical work. Those offered a job can spend a week in the office, five days a week in person, and Gupta offers to help them interview elsewhere, hoping they still choose Boundary.

2025-02-03 · open the original ↗

• Coval started with voice agents because they are constrained and booming. Hopkins says one person talking to another is a narrower medium than self-driving or web agents, so better metrics and workflows can be built for it; voice agents were also "exploding" at least six months before recording, and they extend call centers and phone trees companies already run.
• Voice gives any two businesses a universal natural-language API. She argues a company's agent can expose what it knows, such as claims data, appointment availability or opening hours, to any caller, whether that is a person on a traditional phone without an API or another agent, without integration maintenance.
• Hopkins predicts voice is the first big platform since mobile. Citing a Steve Jobs talk on how early TV filmed plays and early mobile shrank websites before native forms emerged, she expects every company to be expected to have a voice or chat agent, and new experiences that mix speech with web forms, for example for delivery drivers, police officers or truckers away from their computers.

2025-02-02 · open the original ↗

• Even near-human agents will need monitoring at scale. Hopkins says Coval's vision is to manage and understand how agents behave at scale even once they execute tasks exceptionally well, just as call centers and large companies still review human performance; understanding agents is what lets society reason about, legislate and deploy them for the right uses.
• Hopkins expects human creativity to build on agents, not yield to them. She predicts that 50 years from now will be as different as 100 years ago is from today, but that change is neither as dramatic nor as linear as it seems; if agents take over mundane work like spreadsheets and email, she argues, people will become more creative and connected, as they did with computers.

2025-02-01 · open the original ↗

• Agent quality is a spectrum, so Coval layers many metrics. Hopkins says no single score separates good from failing conversations; a suite of metrics shows trends and allows trade-offs, because getting the most-used case wrong, such as booking an appointment, matters more than a 99% average, unlike traditional machine learning.
• Combining metrics separates real failures from false alarms. She says a short conversation isn't bad on its own, but one that is short, missed its goal and skipped required steps is; filtering by metrics directs scarce human review to the most interesting examples.
• Workflow and function-call checks catch technically completed but wrong tasks. Coval checks conversations against JSON-described workflows, for example an appointment booked without asking for the phone number, infers the workflows agents actually follow in production, and verifies that tool calls were made with the right arguments.
• Reference-free evaluation suits non-deterministic agents better than golden datasets. Hopkins says reference-based metrics need curated expected outputs that must be maintained as prompts, models and infrastructure change, while reference-free ones infer the correct answer from context; borrowing from self-driving, Coval also compares agent and human task times, where much longer times plus repetition suggest an agent going in circles.

2025-01-31 · open the original ↗

• The podcast is hiring an account executive to sell sponsorships. The host says the role is open only to applicants based in Canada, ideally near the Eastern Time Zone, with at least 5 years of full-time sales experience, and owns revenue targets and sponsor strategy, with a technical AI or data science background a plus.
• The host says the show is far below its revenue potential. He states that podcasts with comparable listenership earn about 10x its revenue, after more than eight years, over 850 episodes at 104 a year, and three years of sponsorship. He suggests the high-production TV ads he began hosting in the past year for Nvidia, Dell and AT&T could become a sponsor offering.

2025-01-30 · open the original ↗

• Agent errors compound across pathways, so evaluation must be multi-step. The host poses a triage agent calling the wrong tool early; the Coval guest says multi-step evaluation differs from single LLM calls because non-determinism multiplies across every possible path, though agents sometimes recover on their own.
• Self-healing and redundancy give agents graceful failure. She describes a cheap, low-latency model answering while a slower "overthinker" reviews the whole conversation and prompts it back on track, plus fallbacks borrowed from aerospace and self-driving. She says the majority of Coval's customers can already hand off to a human when a task is too complex.
• Reliable agents can be built from unreliable parts, as cloud was. She points to servers being unreliable yet cloud systems reaching six nines through redundancy and fallbacks. The host goes further, calling agents handling complex tasks "inevitable"; she adds that each reliable enterprise deployment raises the tide for all, as with self-driving.

2025-01-29 · open the original ↗

• The workflow turns manual review into automated metrics. The Coval guest describes building an MVP, iterating on the prompt without voice, then running test sets of scenarios (book an appointment, issue a refund, complain about an airline experience) as 100 simulated conversations at once. Failures get reviewed by hand, what the developer looks for becomes a metric, fixes such as less eager interruption get re-simulated, and the loop then moves to production monitoring.
• No voice-agent platform fits every use case. She declines to name preferred providers, saying Coval aims to stay framework-agnostic so customers can switch as prices and requirements change. The axes she gives are low-code versus configurable (open-source orchestrators offer more control over function calling and RAG) and the weight placed on instruction following, function calling, workflow following, naturalness and creativity.

2025-01-28 · open the original ↗

• Self-driving simulation methods transfer to testing voice and chat agents. Hopkins, who led evaluation job infrastructure at Waymo before founding Coval, argues that agents, like self-driving cars, must respond to a changing environment, so they need many simulated pathways rather than manual back-and-forth testing, which she says is what most companies do today. Coval simulates scenarios (e.g. "book an appointment for tomorrow") 10 or 100 times, and re-simulates from production transcripts the way Waymo replays road logs.
• Test coverage is a cost-and-latency trade against signal. Hopkins says Coval works with customers to decide what runs on every PR, every six hours or nightly, and before big releases, and to check whether test sets resemble the distribution seen in production.
• Layer reference-free metrics rather than chase one score. She describes workflow-adherence checks against a JSON workflow, function-call checks inferred from the conversation rather than a golden dataset, unanswered-question detection, and agent-versus-human task time. A short call is not bad on its own; a short call with an unmet goal is. Getting the single most common case wrong matters more than a 99% average.
• Reliability can be built from unreliable parts, as cloud infrastructure was. Hopkins cites redundancy, fallbacks, a slow "overthinker" model monitoring a fast low-latency one, and human handoff, which she says most of Coval's customers already have. She judges self-improving agents and automated prompt optimization still too early; visibility beats the time saved, for now.
• Voice becomes a universal natural-language API between businesses. She says Coval started with voice because it is more constrained and was exploding faster than other agents about six months before recording, and predicts voice is the first platform since mobile every company will be expected to have.

2025-01-27 · open the original ↗

• There is no solid ground for workers facing AI. The host premises that remote keyboard-and-microphone jobs, including much software and data science work, are easiest for AI to replace. Azhar tells listeners to feel apprehensive, and to spend two or three days throwing their hardest problems at LLMs until they are shaken, then "come out the other side".
• Jobs already turn over fast, and new ones keep appearing. He cites that 53% of Gen Z want to be influencers, 14 years after his own influencer-ranking startup, and economist David Autor's finding from about 80 years of US data that 60% of today's jobs did not exist in 1940. He notes the tension between middle-class desk-job growth and lower-paid service jobs over 30 to 40 years.
• Work is becoming "boss-like", and skilling up is the only preparation. Azhar says he and his team now set goals for teams of one or more AI systems, where domain knowledge and judgment help. He expects most jobs to change, some to vanish and new ones to appear, but says nobody knows how fast, and that even in 20 years the question may be open.

2025-01-26 · open the original ↗

• Energy is shifting from a commodity to a technology that cheapens. Azhar argues AI's energy demand will catalyse discipline in building the system. He says coal, oil and gas have not got cheaper in 200 years, while solar panel costs fell by, he thinks, 99% over 20 years, with batteries following the same exponential pattern and intermittency "a completely solvable design problem".
• The transition will mirror telecoms giving way to the internet. He likens the fossil system to the old phone network, reliable but needing about a billion dollars to enter, and forecasts a decentralised, cheaper system led by solar plus batteries, with wind, nuclear and maybe fusion. He puts the energy transformation at a couple of decades and cheap carbon capture and desalination at decades.
• The pessimistic case is not grounded in evidence, he argues. Azhar says peer-reviewed papers in Science and Nature show solar needs less material than fossil fuels, that lithium is plentiful but under-extracted, and that an Oxford study from about two years earlier found a fully renewable system cheaper to run, saving "trillions" the faster it is built. He likens the objections to a UK mobile CEO's 1999 vow never to allow online bill payment.

2025-01-25 · open the original ↗

• When quantum computing becomes useful is genuinely uncertain. Azhar says he does not know, and that quantum builders are scientists who see the world "as it could be", so their time frames differ from product builders'. He expects amazing things once machines have tens of thousands to millions of logical coherent qubits, and notes Google announced its random-number benchmark breakthrough twice.
• Transformers on GPUs may get there before quantum machine learning. He argues transformer architectures are "eating" protein, materials and robotics work, so the overlap with quantum machine learning may be smaller than thought. His one-sentence view is to keep building quantum computers, but that there is good reason to spend time on the current AI wave.

2025-01-24 · open the original ↗

• AI engineer tops the list of fastest-growing US jobs. The host relays LinkedIn Economic Graph research on millions of jobs started between January 2022 and July 2024, which excluded internships, interim and student roles and jobs dominated by a few employers. He notes it does not disclose its minimum-count threshold. US AI engineers' top skills are LLMs, NLP and PyTorch, and they cluster in San Francisco, New York and Boston. The role is 36% fully remote and 27% hybrid.
• AI consultant and AI researcher also rank among the fastest-growing. AI consultant is second and AI researcher twelfth. The host says the consultant role is less technical (top skill prompt engineering, top prior role operations associate). AI researchers' top skill is deep learning, and only 11% are fully remote and 19% hybrid.
• AI roles are leading growth abroad too. In the reports AI engineer is first in the UK and Netherlands, fifth in Sweden, sixth in Canada and Israel, and twelfth in India. AI researcher is third in Canada and Israel, ahead of AI engineer there, and ninth in the Netherlands.
• The host reads this as data science splitting into specialties. His inference is that data scientist has ceded its high-growth spot to AI-specific titles, not that its skills matter less. He notes data scientist is among the most common roles people move from into AI engineering and research.

2025-01-23 · open the original ↗

• Dictating a day's jumbled tasks to a chatbot yields an organised plan. After the school run Azhar records up to 10 minutes of mixed tasks in Claude's audio mode (a speech, car insurance, a parking ticket, a Stripe refund), and on reaching his desk 15 minutes later finds a sorted to-do list, a drafted appeal letter and ordered speech notes. He calls Claude 3.5 Sonnet good enough to change his day.
• The missing piece is turning plans into actions. For now he copies and pastes; he says AI companies are promising that step "this year". He expects hundreds or thousands of agents working for each person and wants a "chief of staff" supervisor agent that takes unstructured input, doubting Apple will build it.
• His multi-agent debate succeeds about half the time. He defines a historian of technology, an investor and a cynic, run on o1 and at least two Claude 3.5 models, with an orchestrator making them argue. A run takes five or six minutes and costs about 20 cents, and fails about half the time, which he says matches human teams on open-ended research. He builds it in Wordware, disclosing he is an investor, or Lindy.
• Learning AI means working close to the API. Azhar says calling models directly lets you set temperature (sensible but creative, around 1, for an evaluator), which third-party interfaces hide. He also uses Gemini Advanced research, NotebookLM, and an email-drafting tool he has invested in; the host describes using You.com's research mode alongside Claude 3.5 Sonnet.

2025-01-22 · open the original ↗

• Linear forecasts ignore how things actually grow. Azhar says business plans still show fixed increments, though growth follows a logistic S-curve with an exponential phase; "nothing is linear except our birthdays". He points to compound interest and late pension saving, and says Bay Area tech internalised Moore's Law. Listeners have lived through exponentials in phone upgrades and data lakes growing from a gigabyte to a petabyte.
• Model growth as compounding percentages with feedback loops. He says he has never used linear increments in more than 20 years of planning, and builds in feedback loops such as Intel chips enabling Windows features that drove upgrades. A linear multi-year S3 storage budget, he argues, is the extreme assumption, even if a realistic one alarms the CFO.

2025-01-21 · open the original ↗

• Assuming exponential compute growth stops is the more radical bet. Azhar calculates world computing capacity has grown about 65% a year compounded for 52 or 53 years since 1972, and his baseline for 10 to 30 years out is to extend that line. He grants the turkey-before-Thanksgiving objection but argues the curve rests on processes (parallelism, interconnects, financial incentive, soon AI help) rather than on shrinking transistors alone.
• Linear forecasts are the extreme assumption in planning. He says he has modelled growth as compounding percentages with feedback loops (Intel chips enabling Windows features driving upgrades) for more than 20 years, and that a linear S3 storage budget contradicts history. Most real growth follows a logistic S-curve; "nothing is linear except our birthdays".
• AI-native companies are opening an "exponential gap". Azhar says fast-growing AI companies reach $30 million annualized revenue in about 20 months against about 60 for SaaS, and that Anthropic reached a billion dollars in sales in
  2024. His advice is to build "the muscle of working with AI" on APIs and semi-finished tools, neither finished SaaS workflows nor your own foundation models.
• Energy is turning from a commodity into a cheapening technology. He says fossil energy has not got cheaper in 200 years while solar panel costs fell by, he thinks, 99% over 20 years, with batteries on a similar curve, and likens the shift to telecoms giving way to the internet. He cites an Oxford paper from about two years earlier finding a fully renewable system cheaper to run, and puts the energy transition at a couple of decades and cheap carbon capture and desalination further out.
• AI is needed to keep knowledge growing as population peaks. Azhar expects human population to peak in 60 or 70 years, after which fewer people can work on problems. On jobs he says there is "no solid ground": citing David Autor's finding that 60% of today's US jobs did not exist in 1940, he expects most jobs to change, everyone's work to become more "boss-like" directing AI, and skilling up as the only preparation.

2025-01-20 · open the original ↗

• AWS's generative AI stack runs from plug-and-play to fully granular. The guest (Eremenko, in episode 853) says Amazon Q offers no choice or customisation of the base model. Q Business is effectively RAG across S3, RDS, Kendra, Gmail, Dropbox, Slack, Zendesk, Jira and Salesforce, with an option to fall back on the model's own knowledge. Q Developer is a coding assistant in JetBrains, VS Code and the CLI, and an AWS-account assistant.
• The middle layer suits most; the granular one needs technical staff. He describes Bedrock as pay-as-you-go access to proprietary and open-source models, with prompting, RAG and agents. SageMaker covers the whole machine learning pipeline with finer customisation and deployment, but needs more technical staff or a partner. The guests disclose no Azure or GCP experience, after over two and a half years on AWS.
• A low-code tool outperformed his hand-tuned models, he says. The second guest (de Ponteves) reports that SageMaker Canvas built, trained and tuned a model in a few clicks and five minutes that outperformed every model he had spent hours tuning on his usual benchmark dataset. He also praises Data Wrangler for preprocessing and JumpStart for vision and NLP foundation models beyond LLMs.

2025-01-19 · open the original ↗

• Instruction-based fine-tuning adds an explicit instruction to each example. The guest (de Ponteves, in episode 853) fine-tuned an Amazon Titan model on niche topics with an instruction to answer in two or three words ("a dog" to which pet is more attention-seeking), and says the result gave concise, straight-to-the-point answers. The difference from domain-specific fine-tuning, he says, is an extra dataset column carrying the instruction. The host notes his former company Nebula instead did domain-specific fine-tuning of Llama models with LoRA.

2025-01-18 · open the original ↗

• Start with cost, modality and the customisation the model supports. The guest (Eremenko, in episode 853) says pricing differs between models, that input and output types (text, image, video) rule models in or out at once, and that you need to know which customisation methods you need before checking which models support them.
• Match inference options and latency to how the model is used. He contrasts real-time uses with no delay (an in-game feature, a natural voice conversation) with asynchronous answers in five minutes and background batch jobs.
• Benchmarks, language and size trade performance against price. He notes that leaderboards score the base model before you customise it, that average performance at a lower price can suffice where accuracy is not business-critical, and that small language models are gaining ground.
• Scalability, compliance, licensing and energy are easily overlooked. He asks how a deployment serving 10,000 users grows to 100,000, flags growing AI regulation and model licences, and suggests weighing a provider's chips and energy use in training and inference.
• In practice, dataset format decided the model. The second guest (de Ponteves) says he chose an Amazon Titan model for one fine-tuning experiment and a Llama model for a medical chatbot because each matched his data's format, and mentions Amazon's just-released Nova models.

2025-01-17 · open the original ↗

• Intelligence began with physics and chemistry storing information. The host relays Ray Kurzweil's first epoch from The Singularity Is Nearer: atoms formed a few hundred thousand years after the Big Bang, then molecules, with carbon's stable chains allowing elaborate information. Self-replicating RNA, though not alive, let natural selection act. The host adds his own awe that physical constants are fine-tuned enough for this.
• Biological life is the second epoch. Per Kurzweil, RNA learned to make proteins and became trapped in fatty-acid bubbles to form protocells. Selection for more stable membranes and storage, eventually double-helix DNA, produced single-celled and then multicellular organisms several billion years ago.
• Technology, beginning a few thousand years ago, is the fourth epoch. Humans began storing information outside their brains, from clay tablets to solid-state drives, and now machine learning that perceives and reasons. The host forecasts that simply scaling parameters, inference compute and data will keep machines passing humans on harder tasks, and that within a few years scientific discovery may be led more by machines than humans.
• We are entering the fifth epoch, merging brains with computers. Today brain-computer interfaces are implanted mainly for paralysis or motor loss after spinal injury or stroke. Kurzweil expects ordinary brains to be augmented over coming decades, vastly expanding memory and abstract thought.
• The sixth epoch turns the cosmos into a substrate for computation. The host calls it Kurzweil's most speculative idea: intelligence spreads beyond Earth and restructures matter, limited only by the speed of light and quantum mechanics, until individual intelligences merge. He says it may be wild conjecture, but that Kurzweil has been prescient about AI's pace.

2025-01-16 · open the original ↗

• The first three lifecycle steps are the labs' job, not yours. The guest (Eremenko, in episode 853) says data preparation, architecture selection and pre-training, which costs "hundreds of millions of dollars", are done by companies like Meta, Anthropic, Google and OpenAI. He adds that businesses repeating them would make global warming much worse.
• Fine-tuning narrows a general model to proprietary data. He gives examples of medical journals, legal data, or ten years of customer-service call transcripts. The model learns specific terms and style the way a fluent speaker learns a specialist vocabulary.
• Evaluate against business metrics, not just technical scores. He says technical scores like BLEU, ROUGE and BERTScore are not enough, and a model must also be tested on the business tasks it is meant to do.
• Deployment, monitoring and iteration form an ongoing loop. He explains an endpoint as just a URL with parameters, says models degrade through data and model drift like a car needing maintenance, and says changes in the business, in customer expectations or in legislation send you back through steps four to eight.
• A single AWS service covers most of the lifecycle. The second guest (de Ponteves) says Amazon Bedrock handles most steps except pre-training, with data prep done in other AWS services. His students fine-tuned a Llama model on a Hugging Face medical-terms dataset in "dozens of minutes" and got a chatbot that could discuss advanced terms. The host separately notes that 2017 was the transformer paper, and ChatGPT in 2022 was GPT-3 with RLHF.

2025-01-15 · open the original ↗

• A foundation model is a pre-baked base layer businesses rent and customise. The guest (Eremenko, in episode 853) likens it to a cake's sponge layer, baked by well-funded labs such as Anthropic, Meta and Mistral, to which a business adds its own toppings for its use case. He says the analogy itself came from ChatGPT.
• Foundation models are broader than large language models. The host offers a Venn diagram in which LLMs sit inside foundation models alongside vision models of the kind a Waymo car relies on, and the guest agrees, adding image and video models.

2025-01-14 · open the original ↗

• Businesses should customise foundation models, not build them. Eremenko frames a foundation model as a cake's base layer that big labs bake and businesses rent and top. Of his eight lifecycle steps, the first three (data prep, architecture, pre-training costing "hundreds of millions of dollars") are done for you; a business handles fine-tuning, evaluation, deployment to an endpoint, monitoring for drift, and iteration.
• Model choice turns on many factors, and in practice data format. Eremenko lists twelve: cost, modality, customisation options, inference options, latency, architecture, benchmarks, language, size, scalability, compliance/licensing and environmental impact. De Ponteves says his own picks (an Amazon Titan model, a Llama model) came down to which fit his fine-tuning dataset's format. The host relays Andrew Ng's advice to ignore cost while prototyping; Eremenko objects for enterprises certain to roll out.
• Domain-specific fine-tuning is the business default. Eremenko ranks it the most common training-time method, ahead of instruction-based fine-tuning (which de Ponteves describes as an extra instruction column in the dataset), RLHF, continued pre-training and training from scratch. De Ponteves reports fine-tuning a Llama model on a Hugging Face medical-terms dataset in "dozens of minutes" to get a medical chatbot.
• Inference-time levers give large gains for little effort. The pair cover temperature (de Ponteves's story generator repeated one story word for word at temperature zero), RAG over a vector store (a French-dessert assistant from one PDF), agents that split tasks into steps so a simpler model can beat a bigger one, and prompt templates, which Eremenko warns can still be hijacked by injected instructions.
• Pick an AWS service by how much control you need. Eremenko calls Amazon Q (Business, Developer) plug-and-play with no model choice, Bedrock the pay-as-you-go middle layer, and SageMaker the most granular. They disclose no Azure or GCP experience; their AWS focus follows their course business's certifications. De Ponteves says SageMaker Canvas beat his hand-tuned models on his benchmark dataset in five minutes.

2025-01-13 · open the original ↗

• Using quantum computers does not require a physics degree. Neukart (Terra Quantum, in episode 851) says physicists are needed only to build chips. He argues the real barrier is meeting superposition and entanglement first as adults, and suggests quantum games for children. For anyone in data science, he calls quantum "a technology like GPUs" that can be added to the stack.
• Regression, classification and LLM hybridisation are fertile ground. He names image classification, chassis-damage inspection on production lines, medical images and self-driving, with LLM hybridisation an active research area with open questions. A model can be trained with a quantum computer and still run inference on a mobile device without one. For learners he points to Terra's TQ42 platform and TQ Academy training.

2025-01-12 · open the original ↗

• The company funds medical work it says it does not need commercially. Neukart says Terra Quantum is about to start clinical trials, paid for itself, of laser treatments for osteoarthritis and arteriosclerosis. A fibre-optic channel delivers circular laser radiation, with imaging and laser parameters optimised by a hybrid quantum algorithm. He claims the approach could prevent stroke and heart attack.
• Some results are released without a customer. He says Terra's hybrid quantum classifier for fatty-liver (steatosis) cases outperformed the benchmark and was published for anyone to use. The host's framing cites the CEO's meeting with Pope Francis on AI and quantum for human well-being.

2025-01-11 · open the original ↗

• Quantum supremacy headlines compare against problems built for quantum. The host cites Satya Nadella's claim of a billion years classically versus under two minutes quantum for RSA-2048, and Google's Willow claim of five minutes versus "10 septillion years", calling such comparisons unfair. Neukart (Terra Quantum, in episode 851) agrees real-world problems matter more, but says RSA is a real one because quantum computers can find the prime factors of very large numbers efficiently.
• Quantum-safe security combines three layers. Neukart describes post-quantum algorithms, citing NIST's recently released standard of three algorithms, which cannot be cracked efficiently "as of current understanding". Quantum key distribution over fibre encodes keys in photon phase or polarization so eavesdropping is detected. Quantum random number generators, using photons or avalanche breakdown, cannot be reproduced the way seeded pseudo-random generators can. Combined, he says, "you're absolutely secure".

2025-01-10 · open the original ↗

• Use the best model first and optimise cost only after success. In episode 841 Andrew Ng says a full weekend day of his own coding ends at about a $5 OpenAI bill, and agentic workloads at most tens or low hundreds of dollars. Cheaper models or supervised fine-tuning before something valuable works is, he says, "most likely premature optimization".
• AI engineering starts from business requirements and a baseline. In episode 847 Ed Donner says there is no best LLM, only the right one, chosen on data, business-outcome metrics and non-functional limits (budget, time to market). He advises a non-LLM baseline first, then usually closed source. Open source fits large proprietary datasets, sensitive data, inference-cost pressure, or on-device use with small models like Llama 3.2.
• Agentic systems spread data across locations, changing security needs. The episode 843 guest from Protopia says agents will touch on-prem, private-cloud and multi-tenant public-cloud data, so exposure must be managed per component. He points to homomorphic encryption, including a recent Apple announcement for information retrieval, as one piece alongside his company's Stained Glass.
• One company gave 2024 both its biggest flop and wow. In episode 849 the host named OpenAI's o1 his wow moment while his co-guest named OpenAI her disappointment. Her own wow was NotebookLM's human-sounding podcasts, and Waymo and Tesla's full self-driving were sub-wows. She relays a Waymo co-CEO describing a fleet of over 100,000 cars as building "one driver".
• Treating AGI as a coming rapture is a religious frame. In episode 845 Greg Epstein, author of Tech Agnostic, reads the host's singularity hopes as a "slow moving multi-stage Rapture". He argues uncertainty deserves dignity over confident leaders, and that Sam Altman's $7 trillion ask reflects OpenAI burning cash with no route to profit. He suggests there may be benefit in moving slowly and losing the AI arms race.

2025-01-09 · open the original ↗

• Scheduling is intractable classically and suits hybrid quantum. Neukart (Terra Quantum, in episode 851) describes a production problem for Volkswagen: fitting 250-odd test classes, each with subtests, to staff with differing skills for cars leaving the line. The same algorithm can schedule EV software subroutines to cut power use. He concedes quantum also cannot prove optimality, so Terra compares quantum and classical results and keeps the better one.
• Logistics, satellites and finance are other live optimization uses. He cites minimising fleet size and empty miles for perishable goods that must reach hubs within windows such as 1 to 3 p.m., satellite constellations, market prediction, collateral optimization and exotic options pricing. He groups quantum uses as optimization, machine learning and simulation.

2025-01-08 · open the original ↗

• "Hybrid" should mean integration in software, not just cloud access. Neukart (Terra Quantum, in episode 851) says many offerings are a cloud provider bolting quantum processors onto HPC and leaving the combining to the user. Terra instead expresses part of a network, such as a fully connected layer, as a quantum circuit and maps its gates to each device, so users switch chips with a parameter and compare performance.
• Short coherence times force shallow, careful circuits. He says a chip holds superposition only for nanoseconds to microseconds, so deep circuits or multi-qubit operations risk collapse before an algorithm finishes. Unlike a classical computer, you cannot pause and read intermediate results.
• Qubits double capacity, but results are probabilistic. Each added qubit doubles the configurations held at once in a perfect machine. Because a correct algorithm can still return nonsense, a circuit is measured perhaps a thousand times, and the majority answer (say 800 of 1,000) is taken.

2025-01-07 · open the original ↗

• Hybrid quantum machine learning is useful today. Neukart disagrees with an earlier guest's cold water on near-term quantum ML. Terra Quantum's approach expresses only parts of a model, such as one fully connected layer, as quantum circuits and leaves the rest on classical HPC. Its software maps circuits to each chip so users switch hardware with a parameter. Circuits must stay shallow because coherence lasts only nanoseconds to microseconds.
• Optimization wins are measured against the best classical solvers. Neukart describes scheduling tests for Volkswagen across 250-odd test classes, cutting empty miles in logistics fleets, satellite constellations, and finance. Terra's promise is to beat whatever the customer uses, such as CPLEX or Gurobi, or it makes no sense to use it. He calls simulation of molecules for batteries and drugs the hardest problem, with more to come on stronger chips.
• Error is no longer the barrier to scaling, he says. Neukart explains that even a two-qubit fidelity like 99.7% compounds into random results at millions of qubits, and that Google's Willow chip showed errors kept down while scaling. Cooling and engineering remain. The host, for his part, calls "septillion years" quantum-versus-classical claims unfair comparisons, and Neukart agrees real-world problems matter more.
• Quantum-safe security combines three layers. Neukart lists post-quantum algorithms (citing NIST's recent standard of three algorithms, uncrackable "as of current understanding"), quantum key distribution over fibre that reveals eavesdropping, and quantum random number generators that pseudo-random algorithms cannot match. Together, he says, they make you "absolutely secure".
• A machine learning background is enough to start in quantum. He says a physics degree is needed only to build chips, and that regression, classification, image inspection and LLM hybridisation are fertile ground; training can be hybrid while inference runs on a phone. He says Terra's team under its CTO achieved room-temperature, ambient-pressure superconductivity in graphite, which could one day remove the fridge from quantum chips.

2025-01-06 · open the original ↗

• Agentic AI is the top prediction, despite the "2024" title. The predictions discussed are for 2025. The co-guest counts over 600 agent startups and expects specialised agent "brains", like Waymo's single driver. The host reports agentic AI sessions at NeurIPS overflowing, and argues agents became viable because LLM error rates fell, citing o1 checking its work step by step. She cautions that permissions between rival companies' platforms may hold agents back; he adds data security and privacy.
• AI will spread into everyday devices, not always sensibly. The co-guest points to Meta's glasses and real-time translation, a possible Apple comeback, and Microsoft's screen-snapshot AI computer, pulled over privacy concerns. She cites an AI electric toothbrush at Costco as "gone too far"; the host saw golf drivers marketed with "AI" in the name.
• AI-driven scientific research will accelerate, but may dent scientists' job satisfaction. The host cites Sakana's AI Scientist and expects AI to run physical labs. The co-guest cites an MIT paper finding AI-assisted researchers made 44% more material discoveries, 39% more patents and 70% more prototypes; the host adds that the same study found job dissatisfaction rose. Her hypothesis is that specialists feel replaced while general knowledge workers welcome AI.
• 2025 will test whether AI infrastructure spending pays off. The co-guest will watch xAI's supercomputer as the first full test of scaling laws, and expects continued infrastructure investment, especially in energy and nuclear. She recalls Mark Zuckerberg citing roughly 600,000 Nvidia GPUs, unsure of the exact figure.

2025-01-05 · open the original ↗

• The co-guest names open source, and consumers, 2024's overall winner. She cites Meta's Llama as the company-level example. Her reasons are more startups, more capable and cheaper models, and California bills being turned down, which she calls good for open source.
• The host's winner is the maker of his everyday chatbot. He names Anthropic and says he subscribes to Gemini, ChatGPT Plus and You.com but uses Claude at least 80% of the time. Claude 3.5 Sonnet handles his neural-network debugging and podcast summarising without needing o1's power, and he likes its "warm" interface. He still uses Gemini for very large files. The co-guest says Perplexity has nearly replaced search for her over the past two months.

2025-01-04 · open the original ↗

• One model was the host's wow and his co-guest's disappointment. The host picks OpenAI's o1, the company she named as her disappointment. She reads the split as a sign of "TikTok-ified" expectations in AI, where anything that does not wow weekly gets written off.
• The co-guest's wow was an AI podcast tool that sounds human. She says NotebookLM's generated hosts felt like a real two-person podcast. The host had nearly picked it too: his episode 822 aired a 12-minute NotebookLM episode about his PhD dissertation, which he says drew some of his most-commented posts of the year.
• Self-driving was the shared "sub-wow". The co-guest praises Tesla's full self-driving. The host calls Waymo a higher level of autonomy, with no one behind the wheel, in San Francisco and Scottsdale at recording, and predicts disruption of truck driving, which he calls the top job in most US states, and of roadside businesses. He still expects automation to raise employment, as in the past. The co-guest relays a Waymo co-CEO calling a fleet of over 100,000 cars "one driver".

2025-01-03 · open the original ↗

• A continuous calendar shows real time between dates at a glance. The host releases his 2025 version, a Google Sheet with one row per week and one column per weekday, so dates are not split by week or month boundaries. Six months fit on a page, and US federal holidays are marked in red.
• Pursue big goals through tiny daily behaviour changes. The host recommends James Clear's Atomic Habits and paid tiers of Claude, Gemini 2.0 or ChatGPT for turning open-ended brainstorming into action items. His own 2025 goal is to cut back to only his highest priorities, with the podcast's 104 episodes a bigger priority than before.

2025-01-02 · open the original ↗

• The co-guest's disappointment of the year rests on unmet expectations. She names OpenAI, saying it set expectations so high that its year fell short.
• The host finds a phone maker's AI launch more disappointing. He picks Apple Intelligence, calling it "a flash in the pan" and asks who uses it. He argues Apple is in a pickle because it prizes security and reliable experiences, LLMs offer neither, and Apple lacks in-house LLMs capable enough, so requests go to OpenAI's GPT models. With huge revenue and margins, though, he would not be surprised to see Google and Apple still dominant five years on. The co-guest hopes Apple is next year's comeback.

2025-01-01 · open the original ↗

• Both speakers name the same search giant as 2024's comeback. Host and co-guest both pick Google. She cites a late run of releases: Veo 2, Willow, AI Studio, NotebookLM and Gemini 2.0. The host says Gemini 2.0 competes at the frontier across benchmarks, though Google is no longer clearly the front-runner in a now "multi-polar" AI world.
• The early leader is more exposed than the search giant, he argues. The host says of OpenAI that scaling data and weights did not repeat the GPT-3-to-4 leap people expected, though he rates o1's inference-time scaling highly, and many at a San Francisco event voiced disappointment with OpenAI's year. Google's search monopoly, advertising, compute and AI talent let it absorb missteps, he argues. He thinks OpenAI is exposed less to giants than to upstarts like xAI or Anthropic, and unsure of its monetisation.