Shane, who says he leads AI evaluations for the data science team at a legal-tech AI company, argues in this free workshop that AI evals should not stop at quality scores. His framework, the "AI impact chain", links a feature change to AI quality, then to customer value, then to business impact, and he ends on causal inference as the method that turns a customer-value gain into a revenue forecast. Once the relationships along the chain are estimated, an offline eval run that takes minutes can be translated into a predicted dollar impact for investment and prioritization decisions.
Three levels of maturity. Level one is shipping blind: no quality metric, so quality feedback arrives as qualitative complaints from the unhappiest users or as churn. Level two is quality-confident: quality improves offline, but nobody knows what it did to users or revenue. He says most current eval content and courses take people from level one to two, and that level three, tying AI improvements to business outcomes, is rare even at large companies, which reach it for some features only.
Why each link is needed. He groups quality metrics into correction rate (how close the output came to what the user wanted), acceptance rate (the share of outputs accepted as-is) and error rate (precision, recall or F1 on annotated failure categories, which he likens to MLOps debugging). None of these is customer value, which for workflow products he measures as time to finish a task or task success rate. A quality metric can chase errors users do not care about, or perfection when 70% one-shot is enough. Customer-value metrics can mislead in turn: his own long browsing for a pizza peel on Amazon would look like engagement but was confusion. Revenue on its own invites false credit; at Nextdoor, he says, engagement spikes that teams claimed for their launches were repeatedly Texas ice storms.
The worked example. A team decides whether to roll out V2 of a text-to-SQL AI analyst, sold at $200 a seat a month and live for 5% of users. V2 adds a planning step, schema context and a second agent that validates and repairs SQL, at higher compute. Latency rises from about 30 seconds to two minutes, but re-asks fall (from about 2.5 per query in V1) and edits per query drop from 2.4 to 0.3, an 87.5% reduction. He posits quality splitting by segment, better on customer-support questions and worse on finance, so the two teams would disagree on shipping. Customer value breaks the tie: time from question to chart export falls from 12 minutes to 3. Repeated A/B tests, he says, eventually show with statistical confidence which quality metrics drive customer value, though he insists AI quality metrics remain the guardrail before anything reaches customers.
From customer value to revenue. Because business impact lags by months or quarters, teams ship on customer value and use business impact to decide where to invest. A/B tests are the gold standard of causal inference, but regression-based methods such as difference-in-differences or propensity score matching also work. In his explicitly made-up example the model yields five extra seats per account per minute saved; 9 minutes saved gives 45 seats per account, 4,500 seats across 100 accounts, $900,000 in added MRR and about $10.8 million ARR, with an interval of roughly $8.6–13 million. That coefficient, he says, is the crucial number; the rest is arithmetic.
Questions from attendees. With too few users or a risk-averse industry he suggests backfilling about six months of quality metrics and running causal inference instead of, or ahead of, an A/B test; with no users at all, focus on quality and qualitative research. When leadership cares only about latency and cost, agree guardrails up front and reframe later debates around ROI. Structured output suits completion and acceptance rates, he says, while chat and multi-step agents need more error analysis. On tooling he finds eval platforms good at traces and scaling judges but weak for annotation, and recommends vibe-coding annotation UIs that mirror the product, such as redline diffs or a live chat pane. He names instrumentation, agreeing on what customer value means and annotation tooling as the main bottlenecks, and says his own pipeline took three or four months before it paid off.
Reviewing five founder-submitted websites on this Design Review episode, Framer co-founder Jorn van Dijk keeps returning to one throughline: clarity about what the product actually does beats polish, and small unresolved details erode trust in everything else on the page.
Lumari, an AI procurement platform. Van Dijk says the heavy animation and vague icon graphics never anchor what the product looks like, and that its two "book a demo" CTAs ask for too much commitment before a first-time visitor understands the value; he recommends screenshots or a short walkthrough video instead.
"A lie," an AI presentation tool. He credits it with showing glimpses of the real interface rather than pure illustration, but flags that its highlighted customer quotes are too generic to back up a "high quality" claim, and that forcing signup before letting someone generate a presentation is a mistake given it's competing directly against a free incumbent, Google Slides.
Juicebox, an AI recruiting platform. Van Dijk likes its distinct visual identity and recognizable customer logos, but says its brutalist black-and-white style with a "win the talent war" tagline feels emotionally disconnected from being a product about finding people, and that its interactive demo hits a signup wall before delivering a payoff.
Leaping AI, a voice-AI call center replacement. He praises its live phone-call demo as genuinely persuasive, but notes the hero imagery of a single phone with one caller reads as consumer-facing even though the copy explicitly targets large enterprise call centers — a mismatch he says undercuts the positioning.
The Hog, a GTM automation tool. This gets the sharpest critique: unclear naming (an unexplained acronym), inconsistent spacing and typography, and testimonials with fully capitalized sentences and no attached company name, which together read as fabricated and cast doubt on the product's other claims. Van Dijk's closing advice, given the page's sprawl, is to strip it down to only what explains the product, test that stripped version on twenty people unfamiliar with it, and expand from there once comprehension is confirmed.
In this episode of the Data Neighbor podcast's series on the analyst's new toolbox, the founder and CEO of LiveDocs (whose name the captions do not settle) argues that the bottleneck in data work is context, not code. LLMs make it trivial to go from raw data to a chart, so the founder says the product's job is to be an "AI data scientist": first an assistant that does 80% of the work in seconds, then a proactive agent that proposes and runs analyses itself, always with a human reviewing. The episode ends on the skills this demands: prompting with the right context, and reviewing output against the objective.
From BI to an AI-native workspace. The founder, a mechanical engineer by background rather than a data person, began with a no-code BI tool for non-technical users. Users kept needing live numbers, then transforms, then SQL they would not write, then a semantic layer and Python. When LLMs began writing code, the trade-off of BI (rigid workflows in exchange for ease of use) stopped being necessary. The founder credits being an outsider with the freedom to question defaults, such as making users pick a query engine when DuckDB is the obvious choice for a CSV.
What "AI data scientist" means. The core problem, in the founder's account, is that business context and technical skill rarely sit in one person; an LLM needs the same context a human analyst would. Stage one removes "yak shaving" (environments, credentials, cleaning) so people work on the business question, whether they are data scientists or data-literate non-coders. Stage two, which the founder says is being built now, makes the agent proactive: suggesting a retention analysis after a churn one, pulling models from Hugging Face, sending pull requests to a dbt repo. The founder predicts some ingestion incumbents will not exist in two years, since an LLM can write a one-off ingestion script.
The demo. In the founder's description a LiveDocs doc is a notebook richer than Jupyter, mixing SQL, Python, charts, inputs and LLM prompts, connected here to ClickHouse, Snowflake and Postgres. Asked for the top-performing branch in a CRM export, the agent made a git commit, sampled the data, wrote SQL, charted the answer and suggested follow-ups, saving everything as plain Python. A slider input drove a reactive query and chart published in app mode, and a notebook was scheduled for 9 a.m. with email, Slack or webhook delivery, with one notebook able to trigger another.
Correctness and handoff. Asked how accuracy is checked, the founder says failed queries are easy, caught by dry runs and retried, while a query that measures the wrong thing needs a human with context. LiveDocs takes uploaded documentation and glossaries, and an explain-the-query feature was in a sprint that week. The founder wants handoff to work like a merged pull request, stating sources and reasoning so the tribal knowledge persists. Indexing gives a coverage percentage of sources the agent has studied, which the founder says roughly correlates with how well it performs.
Skills. Prompting matters most. Counterintuitively, pasting the schema and data types wastes context because tools infer them at runtime; describing table relationships helps more. Choose models by task and treat output with a pinch of salt, at least for the next year.
Pandey, head of Outshift, Cisco's internal incubator, returns from episode 941 to argue that artificial superintelligence will be reached by connecting many agents and humans into flat "multi-agent human societies", not by building ever-larger single models. He sets out a three-layer framework for this, distributed artificial superintelligence: semantic protocols for shared intent, a cognitive memory fabric for shared knowledge, and cognitive engines for shared innovation. He sets it out in a new white paper and links it to Cisco's open-source work. Cisco is among the episode's listed sponsors.
Defining the target. Pandey sorts definitions from OpenAI, DeepMind, Yann LeCun and Demis Hassabis into two buckets. The economic one is a system of agents doing 100% of a human task with no human intervention. The technical one is an objective-driven reasoning engine inventing beyond its training data. Outshift aims for both through scale-out.
The evolutionary analogy. Humans scaled intelligence vertically for 300,000 to 400,000 years. Then a cognitive revolution around 70,000 years ago brought language, and with it shared intent, shared knowledge and shared innovation. Pandey recommends Yuval Noah Harari's Sapiens for the role of shared fiction in that. AI, he says, has only built bigger individual geniuses and now needs an "agentic cognitive evolution".
Shared intent. His running example is two agents building a network for Swisscom, one optimizing the network and one minding compliance, security, budget and schedule. Today they can pass messages but not share meaning. Semantic protocols are meant to handle grounding, discovery of missing information, resolution of conflicts, coordination, negotiation and evaluation. On the substrate, Pandey says natural language is the common denominator while humans are involved, and still is among agents from different vendors; vector space would win only under universal standardization. He concedes that world models suffer from tokenization, but says Outshift starts with language because heterogeneous brownfield enterprises are where Cisco plays.
Shared knowledge. A private conversation shares intent but adds nothing to collective knowledge. The cognitive memory fabric is meant to fix that: cognitive rather than raw data, persistent, and distributed, with sharing and privacy controls. It grew out of the Swisscom work, where agents alone were "a recipe for disaster" and deterministic code alone explored too little. Pairing agents with a network digital twin worked, and after three or four use cases the team is generalizing it across working memory, ontologies and knowledge graphs.
Shared innovation. Citing Raj Reddy's line, via Satya Nadella, that AI can be a guardian angel or a cognitive accelerator, Pandey describes two kinds of cognitive engine. Accelerators speed up tasks. Guardrails keep new designs, such as a never-built low-earth-orbit satellite network, within compliance and physical limits. In drug discovery, an LLM plans and an AlphaFold-like model explores. Compliance, effectiveness and manufacturability agents filter the results down to about 10 options for robotic wet labs. Every stage needs all three layers.
Outshift's position. Pandey frames Cisco as a networking, security and observability company that "empowers the pack" in both superintelligence and quantum networking. Message-level problems are being handled through AGNTCY, A2A and MCP, all Linux Foundation projects where Outshift is a member or launch partner. That includes the Agentic AI Foundation, which he corrects to having formed in December. He insists the bigger unsolved problems are semantic.
Tom and Scott, joined by co-host Sol, spend this Q&A-format episode answering viewer and caller questions on a day when overnight futures swung roughly 100 points before recovering — their point being that no news, however alarming, reliably predicts market direction. The episode moves through crypto and metals structure, options mechanics, and a mild disagreement over how 2026 plays out, closing on the case for high-frequency liquidity providers rather than against them.
Stablecoins, Bitcoin, and physical metals. Asked why Circle trades correlated to Bitcoin, Tom argues there's no structural reason it should — stablecoin buyers want safety, Bitcoin buyers want appreciation, so the correlation is coincidence, not logic; anyone wanting stablecoin-equity exposure should look at Circle, Coinbase or eventually Robinhood. On physical gold and silver, both dismiss it as impractical: dealers reportedly pay 1-5% under spot depending on price level and location, transparency is poor, and the listed futures and ETF market remains the only efficient way to express a metals view.
Mechanics mail. The standard advice to take profits at 50% on defined-risk trades reflects genuine research into an optimal "touch probability," but both say the real rule is to take profits whenever it feels right — psychologically or to free up capital — without overthinking it, comparing it to letting tax consequences dictate a trading decision. On options pricing, an at-the-money vertical spread will trade close to half the width of its strikes regardless of implied volatility or days to expiration, a direct consequence of put-call parity. To sell covered calls against decades of unrealized gains without courting assignment, the fix is simply going further out in time, to a lower delta, and managing the position early — since there's no such thing as intraday assignment, those three levers eliminate the risk entirely.
Defending the bots. Asked how retail can compete with quants and algorithms, both argue retail isn't actually competing with them at all — high-frequency firms hunt arbitrage at a scale requiring hundreds of millions in capital, an entirely different game from selling 45-day premium. Both credit that competition among high-frequency firms for tightening spreads as the reason U.S. markets stay liquid even in stress, calling payment-for-order-flow criticism largely misguided.
A mild, uncertain bearish lean for 2026. Asked which sector could crack the top 10 this year, Scott picks utilities as a defensive, not bullish, call given his broader caution; Tom prefers unloved laggards (citing Nike, PayPal, Zoom-type names) and long-duration Treasuries as a "least worst" trade, explicitly avoiding AI and high-flyer tech. Pressed by a caller on why a lack of historical back-to-back VIX-40 spikes doesn't argue for bullishness instead, Scott reaffirms a modest down-a-few-percent call, citing full valuations and unresolved geopolitical tension, expecting any real weakness to surface in the first quarter if it comes at all. Both voice open skepticism about AI's actual profitability — neither personally pays for an AI subscription despite calling the technology "game altering" — and Tom argues that whatever eventually ends the rally, it's unlikely to be the obvious threat everyone is already watching for.