Shane, who says he leads AI evaluations for the data science team at a legal-tech AI company, argues in this free workshop that AI evals should not stop at quality scores. His framework, the "AI impact chain", links a feature change to AI quality, then to customer value, then to business impact, and he ends on causal inference as the method that turns a customer-value gain into a revenue forecast. Once the relationships along the chain are estimated, an offline eval run that takes minutes can be translated into a predicted dollar impact for investment and prioritization decisions.
Three levels of maturity. Level one is shipping blind: no quality metric, so quality feedback arrives as qualitative complaints from the unhappiest users or as churn. Level two is quality-confident: quality improves offline, but nobody knows what it did to users or revenue. He says most current eval content and courses take people from level one to two, and that level three, tying AI improvements to business outcomes, is rare even at large companies, which reach it for some features only.
Why each link is needed. He groups quality metrics into correction rate (how close the output came to what the user wanted), acceptance rate (the share of outputs accepted as-is) and error rate (precision, recall or F1 on annotated failure categories, which he likens to MLOps debugging). None of these is customer value, which for workflow products he measures as time to finish a task or task success rate. A quality metric can chase errors users do not care about, or perfection when 70% one-shot is enough. Customer-value metrics can mislead in turn: his own long browsing for a pizza peel on Amazon would look like engagement but was confusion. Revenue on its own invites false credit; at Nextdoor, he says, engagement spikes that teams claimed for their launches were repeatedly Texas ice storms.
The worked example. A team decides whether to roll out V2 of a text-to-SQL AI analyst, sold at $200 a seat a month and live for 5% of users. V2 adds a planning step, schema context and a second agent that validates and repairs SQL, at higher compute. Latency rises from about 30 seconds to two minutes, but re-asks fall (from about 2.5 per query in V1) and edits per query drop from 2.4 to 0.3, an 87.5% reduction. He posits quality splitting by segment, better on customer-support questions and worse on finance, so the two teams would disagree on shipping. Customer value breaks the tie: time from question to chart export falls from 12 minutes to 3. Repeated A/B tests, he says, eventually show with statistical confidence which quality metrics drive customer value, though he insists AI quality metrics remain the guardrail before anything reaches customers.
From customer value to revenue. Because business impact lags by months or quarters, teams ship on customer value and use business impact to decide where to invest. A/B tests are the gold standard of causal inference, but regression-based methods such as difference-in-differences or propensity score matching also work. In his explicitly made-up example the model yields five extra seats per account per minute saved; 9 minutes saved gives 45 seats per account, 4,500 seats across 100 accounts, $900,000 in added MRR and about $10.8 million ARR, with an interval of roughly $8.6–13 million. That coefficient, he says, is the crucial number; the rest is arithmetic.
Questions from attendees. With too few users or a risk-averse industry he suggests backfilling about six months of quality metrics and running causal inference instead of, or ahead of, an A/B test; with no users at all, focus on quality and qualitative research. When leadership cares only about latency and cost, agree guardrails up front and reframe later debates around ROI. Structured output suits completion and acceptance rates, he says, while chat and multi-step agents need more error analysis. On tooling he finds eval platforms good at traces and scaling judges but weak for annotation, and recommends vibe-coding annotation UIs that mirror the product, such as redline diffs or a live chat pane. He names instrumentation, agreeing on what customer value means and annotation tooling as the main bottlenecks, and says his own pipeline took three or four months before it paid off.
In this episode of the Data Neighbor podcast's series on the analyst's new toolbox, the founder and CEO of LiveDocs (whose name the captions do not settle) argues that the bottleneck in data work is context, not code. LLMs make it trivial to go from raw data to a chart, so the founder says the product's job is to be an "AI data scientist": first an assistant that does 80% of the work in seconds, then a proactive agent that proposes and runs analyses itself, always with a human reviewing. The episode ends on the skills this demands: prompting with the right context, and reviewing output against the objective.
From BI to an AI-native workspace. The founder, a mechanical engineer by background rather than a data person, began with a no-code BI tool for non-technical users. Users kept needing live numbers, then transforms, then SQL they would not write, then a semantic layer and Python. When LLMs began writing code, the trade-off of BI (rigid workflows in exchange for ease of use) stopped being necessary. The founder credits being an outsider with the freedom to question defaults, such as making users pick a query engine when DuckDB is the obvious choice for a CSV.
What "AI data scientist" means. The core problem, in the founder's account, is that business context and technical skill rarely sit in one person; an LLM needs the same context a human analyst would. Stage one removes "yak shaving" (environments, credentials, cleaning) so people work on the business question, whether they are data scientists or data-literate non-coders. Stage two, which the founder says is being built now, makes the agent proactive: suggesting a retention analysis after a churn one, pulling models from Hugging Face, sending pull requests to a dbt repo. The founder predicts some ingestion incumbents will not exist in two years, since an LLM can write a one-off ingestion script.
The demo. In the founder's description a LiveDocs doc is a notebook richer than Jupyter, mixing SQL, Python, charts, inputs and LLM prompts, connected here to ClickHouse, Snowflake and Postgres. Asked for the top-performing branch in a CRM export, the agent made a git commit, sampled the data, wrote SQL, charted the answer and suggested follow-ups, saving everything as plain Python. A slider input drove a reactive query and chart published in app mode, and a notebook was scheduled for 9 a.m. with email, Slack or webhook delivery, with one notebook able to trigger another.
Correctness and handoff. Asked how accuracy is checked, the founder says failed queries are easy, caught by dry runs and retried, while a query that measures the wrong thing needs a human with context. LiveDocs takes uploaded documentation and glossaries, and an explain-the-query feature was in a sprint that week. The founder wants handoff to work like a merged pull request, stating sources and reasoning so the tribal knowledge persists. Indexing gives a coverage percentage of sources the agent has studied, which the founder says roughly correlates with how well it performs.
Skills. Prompting matters most. Counterintuitively, pasting the schema and data types wastes context because tools infer them at runtime; describing table relationships helps more. Choose models by task and treat output with a pinch of salt, at least for the next year.
Pandey, head of Outshift, Cisco's internal incubator, returns from episode 941 to argue that artificial superintelligence will be reached by connecting many agents and humans into flat "multi-agent human societies", not by building ever-larger single models. He sets out a three-layer framework for this, distributed artificial superintelligence: semantic protocols for shared intent, a cognitive memory fabric for shared knowledge, and cognitive engines for shared innovation. He sets it out in a new white paper and links it to Cisco's open-source work. Cisco is among the episode's listed sponsors.
Defining the target. Pandey sorts definitions from OpenAI, DeepMind, Yann LeCun and Demis Hassabis into two buckets. The economic one is a system of agents doing 100% of a human task with no human intervention. The technical one is an objective-driven reasoning engine inventing beyond its training data. Outshift aims for both through scale-out.
The evolutionary analogy. Humans scaled intelligence vertically for 300,000 to 400,000 years. Then a cognitive revolution around 70,000 years ago brought language, and with it shared intent, shared knowledge and shared innovation. Pandey recommends Yuval Noah Harari's Sapiens for the role of shared fiction in that. AI, he says, has only built bigger individual geniuses and now needs an "agentic cognitive evolution".
Shared intent. His running example is two agents building a network for Swisscom, one optimizing the network and one minding compliance, security, budget and schedule. Today they can pass messages but not share meaning. Semantic protocols are meant to handle grounding, discovery of missing information, resolution of conflicts, coordination, negotiation and evaluation. On the substrate, Pandey says natural language is the common denominator while humans are involved, and still is among agents from different vendors; vector space would win only under universal standardization. He concedes that world models suffer from tokenization, but says Outshift starts with language because heterogeneous brownfield enterprises are where Cisco plays.
Shared knowledge. A private conversation shares intent but adds nothing to collective knowledge. The cognitive memory fabric is meant to fix that: cognitive rather than raw data, persistent, and distributed, with sharing and privacy controls. It grew out of the Swisscom work, where agents alone were "a recipe for disaster" and deterministic code alone explored too little. Pairing agents with a network digital twin worked, and after three or four use cases the team is generalizing it across working memory, ontologies and knowledge graphs.
Shared innovation. Citing Raj Reddy's line, via Satya Nadella, that AI can be a guardian angel or a cognitive accelerator, Pandey describes two kinds of cognitive engine. Accelerators speed up tasks. Guardrails keep new designs, such as a never-built low-earth-orbit satellite network, within compliance and physical limits. In drug discovery, an LLM plans and an AlphaFold-like model explores. Compliance, effectiveness and manufacturability agents filter the results down to about 10 options for robotic wet labs. Every stage needs all three layers.
Outshift's position. Pandey frames Cisco as a networking, security and observability company that "empowers the pack" in both superintelligence and quantum networking. Message-level problems are being handled through AGNTCY, A2A and MCP, all Linux Foundation projects where Outshift is a member or launch partner. That includes the Agentic AI Foundation, which he corrects to having formed in December. He insists the bigger unsolved problems are semantic.
The host's December compilation strings six clips from recent episodes into one argument: AI pays off when it is aimed at measurable outcomes and fitted to how people actually work, and fails when it is adopted for its own sake.
Measurable ROI. Dell's John Roese (episode 953) says Dell's revenue grew $10 billion while absolute cost fell 4% in the fiscal year reported in March, a decoupling he says Dell had not seen in 41 years. He credits picking projects with hard metrics, and cites a consulting report, he thinks McKinsey's, calling generative AI a "BS work detector". The host sets this against MIT NANDA's contentious 95% enterprise-AI failure figure.
AI as coach. Joel Beasley (episode 945) has Gemini measure his stand-up sets against a professional standard of four to five laughs per minute. At recording it was the only browser chatbot that took video. He tracks every set in a spreadsheet, and he built a writing-coach prompt by modeling the feedback at about 10 comedy writers' groups.
Avoiding burnout. A data scientist (episode 952) describes leaving finance in New York for life-sciences data science and hiring a fitness coach after overtraining. Her routine is four gym sessions a week, one gram of protein per pound of body weight, carbs timed around workouts, and at least eight hours of sleep, helped by short phone-free walks.
Getting hired. A Netflix senior data scientist (episode 947) describes a "briefcase" document of a company's pain points and his fixes, which got him into DoorDash. The host doubts it would stand out among today's AI-generated pitches. The guest says Netflix hired him, after earlier rejections, for his rare mix of ads and forecasting from Spotify.
Human context. Sandy Pentland (episode 949) warns that AI systems ignoring human nature could fail as the Soviet Union's did. He argues people are social foragers who take most behavior from culture, that loud voices drive polarization, and that traders who watch other traders do better and weathered 2008.
Workslop. Dropbox's VP of Engineering (episode 951) blames workslop on pressure to adopt AI without clear goals. His fixes are human revision of first drafts, hybrid BM25-plus-vector retrieval for context, internal benchmarks, and training led by high performers.
Olly Hughes, co-founder and CEO of Count, argues on the Data Neighbor podcast's series on the analyst's new toolbox that data teams have been trapped as a support function by tools built for an old problem, and that their real job is to visualize the business clearly and act as its problem solvers. He ends on Count's agent working inside a shared canvas, where analyst, stakeholder and AI break a problem down together and every output stays editable. His closing claim is that data teams already have the superpowers they need, full visibility of the business and problem-solving skill, and that purpose, not tooling, is what keeps them from being replaced by AI.
Origin. Hughes trained as an engineer and started as a consultant inside factories, including a can-making plant in Germany, finding bottlenecks with pen, paper and a stopwatch, then did data work at Tesco and at startups. Working as a consultant he found Tableau and Power BI were tools "to see a number again and again", not to make a decision, and that clients did not feel the numbers were theirs. Power BI's launch, which he dates to around 2015, and having to learn DAX convinced him and his co-founder, also called Ollie, that dashboards were the wrong answer. Shane recalls telecom dashboards that leaders liked and frontline staff found a barrier; Hughes replies that only the executives lacked visibility.
The service trap. Hughes's hypothesis is that the BI tool is among the most expensive, lowest-ROI software a company buys, because it solves a two-decade-old problem of getting data into the business when companies now have too much. He says that in interviews with 400 data leaders they told him about 90% of their teams' work maintains the status quo, such as accurate sales tracking, and adds no incremental value; the best teams he has seen reach 50/50 or 40/60. He names four traps: drowning the business in information; acting as an answering machine, which AI could make a "Cambrian chart explosion"; retreating from stakeholders; and building things the business cannot see. His test is whether the exec team can name three examples of the data team driving value last quarter. A co-host adds that stakeholders' questions degrade when all they ever get back is charts.
Purpose as the fix. Hughes's remedy is operational clarity, making the business look simple with the least information, which leads stakeholders to ask better questions about bottlenecks rather than about what is going on. Beyond that, the team should be the organization's problem solvers, working in agile iterations with stakeholders instead of a waterfall of spec, model and report, and should track time to decision. Purpose, he says, is also what lets a team say no.
The canvas. Count's interface, in his description, is an infinite collaborative whiteboard like Miro or Figma, running on live data. He showed a metric tree breaking revenue into MRR, churn and upsell, and a metric map of about 60 onboarding-funnel metrics a customer built for an activation project, with SQL cells whose dependents are traced and which can be pushed to dbt or a semantic layer. He says customers also use it as a dbt IDE, and answers the complaint that canvases get messy with "thinking is messy".
The agent. Hughes says Count held back on AI to get the paradigm right first, since AI amplifies whatever it is given. In the demo two agents ran at once: a narrow request for New York sales over time, and an ambiguous question about Chicago, which the agent broke into facets before reporting that Chicago ranks seventh in sales but loses money through excessive discounting, mainly in office supplies. Outputs are editable, the agent picks up edits as context, and conversations are visible to collaborators. Shane adds his view that nondeterministic AI products make data science more important. Hughes stresses teams can work this way without Count.
Ozdemir, an AI entrepreneur and author on his seventh appearance, walks through his 10th book, Building Agentic AI. His position is that building agents is mostly disciplined engineering, not model worship. Decide whether the task needs agency at all. Size the model to the task. Evaluate by the cost of errors. Treat reasoning, quantization and other "magic" as trade-offs to measure. The book is the first in the host's Pearson AI signature series, and the host edited it.
Agents versus workflows. Ozdemir says he renamed the book from something like "Building Applied AI" to be honest that much of it is not pure agents. An agent is an LLM with tools that chooses its path; a workflow is deterministic. Most production systems, such as retrieval plus a fixed prompt, are workflows. To choose, he has clients walk through the existing process, which nine times out of 10 settles it. Few conditionals mean a workflow, a handful of pathways a hybrid, and branching at every step an agent. Modern deployments often start at five to 10 tools, and the order and efficiency of tool calls become hard to predict.
Choosing a model. He writes the book around tasks rather than specific LLMs because models will always change. The durable split is between autoregressive models that generate and autoencoding models like BERT. BERT can do embeddings and classification almost as well as a huge GPT model at around 100 million parameters, though he notes diffusion LLMs already bend the categories. For generative models he gives tiers: under 10 billion parameters (CPU-runnable, fine for RAG or search), 10 to 100 billion (multi-turn agents, especially fine-tuned) and 100 billion plus (enterprise-wide, multilingual). He takes Qwen's range from about half a billion to over a trillion as his example. Long context windows help long-horizon agents only if the model can use them; his needle-in-a-haystack test exposed misses, with Grok the worst in the book and Anthropic much improved.
Evaluation. Metrics depend on the task bucket: free text, multiple choice, embeddings or classification. Precision matters when false positives are costly and recall when false negatives are, as in passing a faulty aircraft part versus discarding a good one.
Hybrids and failure modes. OpenAI's deep research always asks a follow-up question, a forced node he says OpenAI accepts because skipped clarification wastes more money. Cursor's plan-then-agent modes do the same. Agents fail in unhuman ways: GPT-4.1 ignored a one-sentence instruction to always consult an Airbnb policy tool 5% of the time. On model preferences, the host says Claude best guesses the output structure he wants from minimal context at the time of recording. Ozdemir links that to Anthropic's emphasis on code.
Reasoning and optimization. The surprise that started as a client hunch is that more reasoning showed no consistent link to performance on MathQA and Humanity's Last Exam. Where there was a gain, it was maybe 1 or 2% for 2x the cost. His benchmark took 5 hours for 180 calls, and one model reasoned for three paragraphs to say hello. Quantization, distillation and LoRA buy memory, speed and cost but "definitely not better": a quantized Llama scored a little worse and its candidate tokens diverged, making it a "cousin" of the original. In fine-tuning a Qwen model with speculative decoding, he found he could predict which questions would not benefit, which he likens to how routing between fast and slow models works.
Rajeeva, co-founder and CTO of Acceldata, argues that agents can take over much of enterprise data management, from cataloging and quality checks to repairing broken pipelines. That works only if the AI is given context about data it was never trained on, and it stays bounded: an expert still approves the result. Acceldata, founded in 2019 by four former Hortonworks engineers, has raised over $100 million across three rounds. It runs engineering largely from India, with staff in the Bay Area and Canada and a team forming in London.
Agentic data management. Rajeeva says data catalogs, MDM and curation have improved only incrementally for 15 to 20 years and are mostly forms and clicking through defined workflows, which agents can automate. ADM now has close to 10 agents, such as a data-catalog agent that knows a table's columns, types, refresh cadence and owner. They are distilled from customer conversations because banks, insurers and healthcare firms have different problems. Powering them is the xLake reasoning engine. Just as Google's index lets a chatbot search the web, xLake gives AI tools to query petabytes across Snowflake, AWS, Azure and on-prem clusters, while handling scale and column-level permissions.
Self-healing pipelines. An agent detects an anomaly in logs, clones the Spark code, adds error and metadata context and redeploys the fix to the cluster, replacing the bug-report, fix and CI cycle. Rajeeva says Gemini 3 one-shots about 90% of what you want. He concedes proprietary Informatica, HVR or Oracle stored-procedure pipelines are hard, and predicts pipelines will move from drag-and-drop to code.
Humans at the gate. Autonomy, he says, means doing most of the work unsupervised, perhaps a week compressed into an hour, while experts approve, reject or demand rewrites, as in code review. His MCP demo, which built a GDPR compliance-scoring agent and dashboard, got about 60% of the work done, maybe 80% with prompting. Enterprises are slow to decide but afraid of falling behind, so Acceldata builds for where they will be.
Architecture and sprawl. Data sprawl is legacy on-prem data plus Snowflake or Databricks plus a Google cloud deal, all needing to be found and joined. Unlike metadata-only players, Acceldata deploys a data plane in the customer's environment over mTLS and can validate up to 100 million rows. It builds on Kubernetes and Spark and publishes its ODP data platform as open source. Customers are enterprises such as Nestlé, not startups.
Leading engineers. Rajeeva says a CTO must take the customer's view, because customers need wins to renew a proof of value and product managers focus on features. Engineers, who studies suggest are productive only 3 or 4 hours a day, quit without a mission. He keeps them with engineer-led projects: ODP, an AI agent that fixes vulnerabilities, and an in-house data-center stack replacing VMware and OpenStack. He hires senior people on whether they can build. Coding interviews are now face to face because remote LLM cheating is too hard to detect, though he doubts the approach will scale.
In their fifth annual predictions episode, St. Lawrence, founder of the Human-Machine Collaboration Institute (HMCI), and the host end on a shared view. Scaling has run short of data and clear direction. The next gains will come from specialized models, continual learning, new research and spatial data. Meanwhile intelligence keeps getting radically cheaper. The host is series editor of her new Pearson book, Becoming an AI Orchestrator, and says he urged her to write it.
HMCI and the book. St. Lawrence's TEDx talk urged asking hard questions about merging with AI and the nature of consciousness. HMCI, an independent research firm working toward a unified theory of consciousness, built an intelligent revenue reinvestment (IR2) model. It channels money from energy and data-center projects into local upskilling, research and infrastructure, and she says Nvidia is partnering on a flagship in California, with more cities planned. The book's premise is moving from musician to conductor of AI agents, which she says let HMCI run as a team of three.
Scoring 2025. Agentic AI dominated so thoroughly that audiences now tire of it. The host adds that agents' task length doubles every 7 months at about 50% reliability, pointing to a full workday by end of 2026. AI reached devices such as AirPods' live translation, though she found CES appliances a poor use of it. AI science was mixed: a materials-discovery paper was retracted, but DeepMind's GNoME project impressed. Enterprise monetization remained the issue, with projects stuck at MVP. The host stresses that the same capability now costs about 100 times less than a year ago. A LinkedIn report bore out her call that AI engineering skills would outgrow data science, which she frames as the data-scientist "unicorn" splintering into a family of roles.
The 2025 awards. For wow moment, she picks Nano Banana's accurate room redesigns. He picks Ed Donner's four-agent crew building a working simulated trading platform over a 90-minute lunch at ODSC East. Google is comeback of the year again, for its integration across Search and G Suite and the data users already entrust to it. For disappointment she names enterprise agents, as in her Substack post "Agents of Disappointment", and he names Apple Intelligence. The host also credits xAI's Grok, which he says was statistically tied with Gemini 3 Pro on LM Arena. Overall winners are Nvidia and "vibes", with GPT-5 disappointing and nobody anticipating a next leap. The host notes coding leads because its results are verifiable and its training data can be simulated.
Predictions for 2026. First, specialized AlphaFold-style domain models, since general models have hit data limits. Second, continual and nested learning, after a new Google paper. Third, a return to research: she cites Ilya Sutskever's recent interview as showing labs lack a clear path. The host argues sample-efficient learning, like a toddler inferring an adult's intent, is the missing breakthrough, but doubts it lands in 2026. Fourth, physical AI and spatial intelligence, valued as new data, with 2027 still "the year of the robot". Fifth, AI ops as a new job, akin to DevOps around 2010.