AI Analyst Lab

2026-11-06 · open the original ↗

• Generic answers lose; put yourself in every answer. The presenter contrasts a passable "leader in local commerce, love to bring my analytics experience" answer to "why DoorDash?" with a personal one (a working parent ordering healthy food on bad work days), and frames "why this company" as a real moment with the product, then an observation, then the company's values, never perks, comp or "exploring options."
• Hypothetical questions are behavioral questions in disguise. Asked what to do when a PM wants results early, the principle answer ("balance speed with rigor") is fine. Her own story is stronger: a PM wanted to call a checkout test on day four, she held it two weeks because Sunday was the biggest ordering day, it came back flat, and she now agrees test length before launch.
• Build an "experience database" by talking, then let AI mine it. She recommends recording at least 30 minutes, unfiltered and in breaks (she did 3 hours), covering projects, people friction, mistakes, ambiguity and leadership. AI then turns it into story cards and number cards, maps them to question types and flags missing impact numbers. She advises two stories per type, because onsite interviewers compare notes in the debrief.
• Six delivery failures sink otherwise good experience. They are: principle before example, no numbers, dodging the harder version of a repeated question, a 3–4 minute answer with the result buried, numbers that "wobble" between tellings of the same story, and "we" instead of "I". Her rule is a headline result and number up front and an answer under about 90 seconds, with detail saved for follow-ups.
• A free repo scores mock answers against these failures. The "interview prep with AI" repo from AI Analyst Lab adds a mock behavioral skill with eight question types and a scorer. In the demo it rated a sample conflict answer 4/10: headline at 2:05 against a 15-second target, about 450 words, roughly 15 "we" and no action "I", and the candidate saying "there wasn't really a conflict." She says to aim for 8/10 or above.

2026-11-06 · open the original ↗

• Bad agent answers are mostly a context problem, not a model problem. Rachel of Hex argues the models are now broadly proficient. She treats agents as "very eager interns" whose quality depends on the context they get and how it is structured.
• Context management is a loop, not a one-off waterfall project. She starts from the questions users actually ask the agent rather than a pre-written list of expected questions, then builds context, measures it with evals and ships it. Some customers gate context PRs on passing evals in CI/CD, which she does not recommend as a first step. Raw Confluence, Notion, Slack or Jira content is "inputs, not context," because the AI cannot tell which pages are stale.
• Semantic models are "a dashboard in a trench coat." This is her self-described hot take. Because no one can predetermine every question, semantic models will be one piece but not the linchpin, behind table endorsement, metadata descriptions, markdown "guides" and evals.
• Hex's context consultant skill takes a gap from evidence to a merged fix. It is built around two subagents, a context architect and an eval engineer. In the demo, the Hex CLI pulled the top three suggestions, led by product-level inventory with 15 examples, and noted that two overlapped. It wrote a 59-line "product inventory metrics" guide and an eval suite on a forked context preview that passed 3/3. She concedes there was no baseline run without the guide.
• A vendor should not own your context. On Atlan or a similar catalog versus syncing into Hex, she says either works but you should pick one, because half-and-half setups drift. Shared skills should live in one central marketplace or GitHub location. She says she reads the generated evals herself for at least the first few loops.

2026-10-25 · open the original ↗

• An AI analyst needs the same onboarding as a new human hire. A skilled new analyst still has to learn the product, which reports people use, what the team means by "session", "customer" or "revenue", and what is routinely filtered out, such as test accounts or pre-login activity. The speaker argues AI needs that same context before its analytics answers can be trusted.

2026-10-25 · open the original ↗

• Which session type to count depends on the question asked. Nextdoor has logged-out sessions (lurkers), logged-in sessions, and "quality" sessions with posting, commenting, reacting, event creation or messaging. For the impact of sessions on ad revenue, the speaker counts all logged-in sessions. Logged-out sessions are excluded to tie the count to real users, and quality sessions are not required because users can view ads without engaging.

2026-10-25 · open the original ↗

• When the calculation is missing, ask AI where it lives. Stuck on finding a session calculation, the speaker asks the AI for another source. It points to the query and data model behind the dashboard, and to a named colleague or the analytics engineering team, and both sources go into the context plan.

2026-10-25 · open the original ↗

• A data team's value test is three exec-recognised wins. The speaker, credited in the title as Count's Ollie Hughes, calls it the simplest and most direct measure: whether the executive team can name three examples from the last quarter where the data team directly drove value, and whether they know the team was directly involved in making an initiative work.

2026-10-25 · open the original ↗

• A messy canvas is fine because thinking is messy. The speaker, credited in the title as Count's Ollie Hughes, answers the objection that the canvas gets cluttered by saying a visible representation of your thoughts lowers cognitive load. If it gets you to an answer faster, you can simplify and clean it up afterwards.

2026-10-25 · open the original ↗

• AI answering business questions risks a "Cambrian chart explosion". The speaker, credited in the title as Count's Ollie Hughes, predicts more charts than anyone knows what to do with. The speaker describes a vicious cycle: confused stakeholders ask clarifying questions, which produce more information and more confusion, an attempt to get clarity by producing more stuff that the speaker calls "really damaging".

2026-10-24 · open the original ↗

• Users' fixes to AI output are free context engineering. The speaker says that anyone rolling out agentic analytics should log every correction users make (a wrong date range fixed, a missing segment added, excess segments filtered out) and then ask those users why they made them. The answers become context that makes future questions correct on the first try.

2026-10-24 · open the original ↗

• Start context engineering with one frequently asked question. The speaker's first step is to take a question that keeps getting asked, have whoever would answer it confirm the answer and every rule that applies, and save that somewhere AI can access.

2026-10-24 · open the original ↗

• A semantic layer replaces the colleague everyone messages about numbers. That colleague, the speaker says, knows why finance's version of a number differs from the product dashboard's and why last month spiked after a data migration. An AI analyst connected to the warehouse, or ChatGPT or Claude through a plugin, has no access to that knowledge, which is where semantic layers, context engineering and guides come in.

2026-10-24 · open the original ↗

• Don't accept from AI what you'd reject from a junior analyst. The speaker says asking AI for retention three times gave 9%, 25.3% and 99%, and that an analyst who did the same would anger the team. That, they say, is why context matters and why the lab spends so much time on context engineering and management.

2026-10-23 · open the original ↗

• AI gets the code right and invents the business definitions. Shane, teaching this lesson in the lab's context series run with Hex, says an AI analyst needs what a new hire needs: what "session" or "revenue" means, what is already filtered out (test accounts), and which data to ignore. He defines context engineering as making that knowledge available to the agent, and context management as the ongoing, governed work of keeping it current so a changed definition reaches every person and system using it.
• A context plan runs question, sources, rules, instructions, checks. The question should be written the way a stakeholder would ask it, with its time period, segment, who needs the answer and what they will decide. Each source gets an owner and its known limits, and each definition gets a source and a confirmer, with gaps written down rather than guessed. Shane calls the first three steps human-led and AI-supported, and the instructions and checks (evals that test whether the instructions were actually applied) AI-led and human-reviewed.
• The demo plan turned "why are iOS sessions down?" into next steps. Using a worksheet tool he built that week and a question from his Nextdoor days, Shane picked logged-in sessions over logged-out and "quality" sessions because ads only need a view, and marked the calculation a working assumption. The tool drafted instructions with a trigger for when they apply and three checks: right session type and iOS breakdown, evidence for the cause rather than the trend as proof, and support for any ad-revenue claim or recommendation to pause product work. Its next steps were to have an analytics engineer confirm the session logic, check whether the dashboard splits by device, hand the plan to ChatGPT, run one week against the prior week, and inspect the result against the checks.
• Scaling context is an ownership problem, not a technical one. Shane favours embedded data teams owning context for their domain, and says to start with questions already asked more than three times, since those come with past answers to evaluate against. The co-host argues for one accountable owner per metric, possibly outside the data team, who approves changes and pushes them out. Shane adds that asking a list of owners "is this all good?" gets no answer, whereas an owner who uses the AI analyst daily has a reason to fix their own metric.
• Route agents to the right guide by trigger, and triangulate conclusions. In Q&A Shane says loading too much context still sends agents off course even with real sources, so a CLAUDE.md-style map should point to guides whose headers say what questions they answer, and the agent opens only the relevant ones. For checking conclusions he says human judgment is still needed, using trace review and having several agents answer the same question by different methods (week-over-week, regression, driver segmentation), with agreement on direction raising confidence.

2026-10-23 · open the original ↗

• Frame the question, its audience and its use before analysis. The speaker's example narrows "why are sessions down this week versus last week?" to iOS sessions, names the director of product as the person who needs the answer, and states the decision: whether to pause product work to find out if it is a bug or real lower usage. An AI reviewer then rephrases the answers, and the speaker chooses which changes to keep before adding them to a context plan.

2026-10-23 · open the original ↗

• Context stays current when metric owners push changes. The speaker says nobody answers a broadcast "is this all good?" request, recalling an analytics engineer who kept asking it. When the speaker owned a metric that was changed several times, one measuring the quality of AI suggestions on legal documents, the speaker updated it or had the analytics engineer do so. Responsibility sitting with the actual owner, plus triggers, is what matters.

2026-10-23 · open the original ↗

• Route agents to minimal context through guide descriptions. The speaker says agents can still lose track and answer the wrong thing from too much context even when the sources are real. A question should first hit a map of guides, each headed by a description of the questions it answers, so the agent reads only those triggers and opens only the relevant guides.

2026-10-23 · open the original ↗

• Analysts carry the risk when executives demand chat-with-data. The speaker, credited in the title as Tableau's Nate Nichols, describes executives who have a striking conversation with ChatGPT and tell the analyst to put it on all their data. Analysts feel caught in the middle, because when it returns a wrong number it is their job on the line, not ChatGPT's, and that brings pressure and anxiety.

2026-10-23 · open the original ↗

• LLMs silently resolve ambiguity, so answers drift between asks. The speaker, credited in the title as Nate Nichols, says LLMs are so eager to be useful that, asked "who's my best salesperson?", they pick some meaning of "best" and answer. Tomorrow's answer may differ because it used number of deals instead of total deal value, so the speaker's team works to shake out the ambiguity before answering.

2026-10-23 · open the original ↗

• Only the decision at the end of the data funnel makes money. The speaker, credited in the title as Tableau's Nate Nichols, says companies pay to store, process and visualise data, but only a better or faster decision produces money. The funnel is lossy, and the step from a dashboard to a better decision has historically had a low conversion rate.

2026-10-23 · open the original ↗

• Users wanted their key number fixed, not a dynamic feed. The speaker, credited in the title as Tableau's Nate Nichols, says Narrative Science's Lexio, a natural-language insights product, was first designed like a social news feed that put the most interesting changes at the top. Users instead wanted the number they need always in the same place. The speaker calls this partly psychological comfort and partly real for people making decisions, and the balance between dynamism and stability an ongoing trade-off.

2026-10-23 · open the original ↗

• Generated stories pay off for tiny, invested audiences. The speaker, credited in the title as Tableau's Nate Nichols, recalls a big client that turned little league stats entered by parents into game recaps ("came from behind off a three-run homer"). Each story had an audience of about a dozen people, but they were reading about their own kids and grandkids, and the speaker says that model resonated.

2026-10-21 · open the original ↗

• Hex won't fix a data team unwilling to partner with the business. The speaker, a Hex executive (credited in the title as its CEO), says they have recently told some prospects not to buy Hex. Business demand for "chat with my data" leads teams to look for a tool to throw at users, a set-up the speaker says is often not built to succeed. The speaker argues against a standalone chatbot for the business alongside a separate tool for the data team, in favour of one system that brings both to the table, and invites teams that want that partnership to get a demo.

2026-10-21 · open the original ↗

• Data people are defined by problem-solving now, not syntax. The speaker, credited in the title as Hex's Barry McCardel, says mastery of SQL or pandas syntax no longer marks a data person, and that Hex has cut back the coding tests it used to give in data interviews. What transcends the technical work "that all engineers can do really well now" is a love of puzzles and mysteries and a way of partnering with others to solve them.

2026-10-21 · open the original ↗

• Evals shouldn't be a one-off backtest you never revisit. The speaker, from PromptLayer per the title, frames evals as answering whether the AI product is good and whether the next change will make it better. They describe a common pattern: a chatbot replies in the wrong language based on users' names, the team backtests the last thousand conversations, iterates on the prompt until it is fixed, ships, and never touches those evals again. Their advice is to step away and work out with pen and paper how a human would grade the output.

2026-10-21 · open the original ↗

• LLM judges can mis-score, so route low-confidence evaluations to experts. The speaker, credited in the title as Composo's Dr. Seb Fox, describes an example where a change the judge itself had suggested, adding a name, made its score fall from 90 to 80 out of 100. The speaker says this isn't universal but appears across a surprising range of LLM-as-judge evaluations, so each evaluation should carry a confidence signal, and the low-confidence ones should go to human domain experts. The clip is fragmentary.

2026-10-21 · open the original ↗

• Always keep at least one eval failing. The speaker, credited in the title as Braintrust's CEO Ankur Goyal, says a fully passing suite means there is more work to do. A user complaint should be capturable as an eval within one or two clicks. With a change run on 1,000 or 10,000 examples and only 30 minutes to review, the problem becomes surfacing the examples most likely to show positive or negative changes.

2026-10-15 · open the original ↗

• Judge a data agent by the analytics workflow, not its features. Shane frames the live lab around the lab's workflow (goal, question, metrics, analysis, insight, decision, story, closing the loop). He says that in his experience with agentic systems, AI can carry the execution at every stage, but people still own which questions are worth answering, what a metric means locally, the level of rigour a decision needs, and the decision itself.
• The agent's data profiling caught the planted quality problems. Given a synthetic grocery-delivery database (six tables, uploaded to ChatGPT's work mode with the data plugin), the agent reported that many non-cancelled orders did not reconcile. It advised exploratory use with clear metric definitions, not treating the financial fields as reconciled. Shane praises its review gates before reporting.
• The same revenue question gave three different numbers. Asked last month's revenue in four parallel chats across models, three answered 278,993, one 276,941, and two noted that excluding tax and tips would give 242,684. Most did not state a definition and none stopped to ask for one, which is what Shane says he would want. A regional breakdown gave Northeast growth of 12.1% in one chat and 13.1% in another, though the recommendation was the same in both (back the Northeast, investigate the Midwest and Mountain). His lesson is to run the same question several times to expose unreliable numbers.
• Calculations are strong; business judgment and pushback are the weak points. The co-host says that across earlier runs they saw no calculation errors and that the agent handled traps like Simpson's paradox, but that it does not always state its assumptions and gives way when strong-armed. Shane says that in an earlier session he told it a CFO-reported revenue figure and it readily aligned to it. He warns that it can become a tool for stakeholders to slice the data until it agrees with them.
• Use it for metric alignment and directional exploration, not reported numbers. Its dashboards were generic big-number-plus-charts templates, and neither of Shane's would open, so he would keep a backup before a meeting. He would use it to pull numbers for several candidate definitions of a metric ahead of a team alignment meeting, and for directional exploratory analysis. The co-host would use it for drilling into segments and for recurring reports once definitions are codified.

2026-10-15 · open the original ↗

• A working query can still use a definition your team rejects. Shane, opening the context series with a fictional outdoor gear retailer, says an AI analyst will find a monetary table, write SQL that runs and return a revenue number built on the wrong definition. Context engineering supplies the company's definitions, date fields and exclusions, and where a question leaves a choice open the agent should ask rather than guess. He says the explanations also need a structured home with a record of changes, rather than meetings, personal notes and Slack.
• Start from existing work, then trace it upstream. A finance report's query shows the filters and calculation in use, a refund policy gives business rules, a data catalog describes the records, and transformation and application code explain what was removed or recorded in the first place (test orders dropped before the reporting table, whether an order is logged at attempt or completion). Shane warns that reused queries carry filters for their original purpose, such as a single promotional campaign, and says to ask the owner about anything unexplained.
• Describe table grain, and endorse the sources the team trusts. In his example, a $100 order of a $30 shirt, a $20 bottle and a $50 day pack, joined to its three item rows and summed, reports $300. Shane recommends endorsements (reviewed-by labels or gold/silver/bronze tiers), last-updated dates, and named owners who update context when, say, finance changes a definition, so that stale reports aren't weighted equally with current ones.
• Keep broad context always loaded and specific guides retrieved. Shane separates company-wide conventions, which Hex includes in every conversation as workspace context, from a library of guides pulled in only for the relevant question. A guide called "sales notes" gives an agent nothing to choose by, while "revenue and orders" with a description in the words people actually use does. His sample guide confirms the period and gross versus net with the asker, defines net revenue as excluding cancelled, refunded and test orders, counts each order total once, and ties out to the daily revenue summary, which also gives a rubric to evaluate against.
• Too much context in one file is the failure mode. In Q&A Rachel of Hex says organised context with front matter and proper retrieval lets models skip what is irrelevant. Shane says gaps are found iteratively by running analyses, asking why they went wrong and running evals. On whether reading code assumes the code is correct, he says bad data foundations just produce wrong answers faster, and that executives using AI analysts may be the forcing function to fix them.

2026-10-15 · open the original ↗

• Most North Stars are defaults, not decisions. The co-host, presenting, says metrics usually arrive through a top-down mandate, a dashboard tool or another team, and that asking AI for "a North Star for my company" returns a pick without the trade-offs. In "weekly active users" each word is a decision: how often value arrives, which action counts, and who the unit is. A copied metric that moves regardless of what the team does is a vanity metric.
• Derive the metric from the customer promise in five steps. The chain runs promise, moment, action, unit, cadence. A language app promising a daily habit gets daily active learners. For a small-business hiring platform, weekly active users would fall whenever a customer succeeded and stopped logging in, so roles filled, with earlier predictors, fits better. For a tax app, weekly use fails on seasonality, and the right metric depends on which promise it makes: done right with no stress, every dollar owed, done in one sitting, or money handled all year, which would favour monthly engaged households.
• The demo prompt has AI interview first and choose last. In a plain Claude chat, the prompt has Claude interview the user one question at a time and push back on vague answers. It then researches comparable companies, returns four to six candidates with what each incentivises, how it could be gamed and when it would be outgrown, and picks one, saying what would change its mind. For the fictional Tidy Tax (promise "done right", moment a federal return accepted on first e-file, unit the household, value once a year from January to April), Claude cited Intuit, H&R Block, the IRS and Lemonade, gave six candidates, and chose households accepted on the first submission over the default of accepted returns. When told the CEO wanted next-season return households, its adversarial review still disagreed.
• Claude's "what would change my mind" list is the analysis backlog. The conditions it gave were that first-try acceptance does not predict customers coming back, that rejections are outside the company's control, and that the rate is already near its ceiling. The co-host says a data team can test each of these against historical data before adopting the metric.
• Pair a lagging North Star with leading drivers, and revisit it. The co-host says the CEO's retention preference is a lagging indicator, so the North Star should be timely and broken into drivers: households starting a return, accepted, accepted first time, and returning. The co-host lists four reasons to revisit it: the promise changes (a pivot to handling money all year would make the monthly metric right), the goal is met, the team learns to game it, or it stops predicting retention and revenue. Asked about zero-to-one products, the co-host says to treat the metric as a placeholder and not to get attached to it.

2026-10-15 · open the original ↗

• Ground every answer in your own experience, not a generic one. The presenter, working through a lesson built around a fictional analyst, "Priya", contrasts a generic answer to "Why do you want to work at Nextdoor?" ("I love your mission of building community…") with an example answer about a neighbour's missing dog, found within an hour through a Nextdoor post, tied to a measurable product moment: how fast a post gets a real response. Because AI can write the generic answer for anyone, the presenter says you need to write down your own personal and technical stories for it to work from. The caption text stops after about 700 words, just as the presenter introduces three reasons interview prep goes wrong, so the rest of the roughly 55-minute lesson is not in the source.

2026-10-15 · open the original ↗

• Recommendations die after the meeting, not in the analysis. The co-host, presenting, names four traps: the action item with no owner, metric or date; "we should", where shared ownership means none; "let's monitor it" with no baseline or target, so monitoring never ends; and "by Q3", a deadline nobody can put on a calendar.
• The close-the-loop checklist has six fields. They are a single named decision owner (not a team), a success metric close to the change (not NPS for a hotfix), a dated baseline, a specific target (fewer than 200 payment tickets, not "reduce significantly"), a real calendar check-back date, and a fallback of revert, escalate or pivot. Shane adds that if it isn't written down it didn't happen, and that treating decisions as bets with a pre-agreed fallback heads off finger-pointing when one fails.
• The demo's skill turned a root cause into a trackable plan. The co-host uses the lab's free AI analyst repo, whose close-the-loop skill triggers whenever an analysis in Claude Code ends with a recommendation, on fictional e-commerce data from "Novamart". Claude Code found a June 2024 spike of about 2x in payment tickets, traced it to app version 2.3.0, and found no double charges and no blocked transactions. The skill then asked who owns the decision, by when, and who follows up, and wrote a markdown plan with owners, check-back dates, guardrail thresholds, fallbacks and a confidence score. The co-host was on Opus 4.7, and says the human remains accountable for the plan.
• Validate the action by triangulating methods, not just the number. Shane lists the validation layers built into the system: reviewing the SQL and Python, tying numbers to ground truth where it exists, and sanity checks such as values that should never be negative. Triangulation sends several subagents to test the same hypothesis with different methods (regression, propensity score matching, difference-in-differences), and agreement on direction is what gives confidence.

2026-10-15 · open the original ↗

• A deck's titles alone should tell the story. The presenter contrasts topic titles ("Revenue overview", "Channel breakdown") with claim titles such as "Revenue grew 78% in H2, but December growth nearly stopped", "TikTok drove all of it" and "AOV fell in all six channels". Each slide must pass four questions: the so-what (a finding, not data), the stakes for this audience, two or three pieces of evidence rather than 15, and an ask rather than "continue monitoring". The framework is tool-agnostic: Claude Code reaches Google Slides, or PowerPoint, through an MCP connection or a CLI.
• Have Claude write a deck brief before it builds slides. Starting from a long Google Doc that the lab's larger, paid skills-and-agents repo produced on fictional Novamart 2024 data, the presenter asked for a four-sentence arc (situation, complication, answer, action) and a slide plan for a 20-minute meeting with a VP of growth. Claude said the document led with a performance overview and buried the complication: H2 growth was bought with traffic, largely from TikTok, and the AOV decline was real and probably not a TikTok mix effect. It recommended holding the FY25 channel budget and closing three data gaps, and planned six slides where a straight conversion would have run to 20 or more.
• A critique skill caught a false headline in Claude's first pass. The lab's deck-critique skill graded the built deck C, 6.8 out of 12 across nine slides, scoring each slide 3, 1 or 0 on the four questions. It flagged that slide seven's headline was false (AOV did not fall every month). The presenter calls the first pass a draft, and the fix step had not finished during the session. The presenter attributes the clean, uncluttered charts to chart rules written into the skills, drawn partly from a storytelling-with-data book.
• Check each pipeline step on new data; collapse it once trusted. Asked why the analysis and the deck were two steps, the presenter says they run as one prompt in practice. On an unfamiliar question, though, you stop at each step to confirm the questions and methods before spending tokens, and merge steps only over later iterations. On PII, the presenter says access should be restricted before Claude Code ever sees the data, with a must-pass rule in CLAUDE.md as a backstop.
• The lab dropped Gamma because Claude Code is more flexible. This is the co-host's opinion: the hosts stopped paying for Gamma two or three months before the session because Claude Code can be made more tailored. The co-host thinks the "SaaS apocalypse" narrative is "overstretched by a mile", though some software can't beat general-purpose models.

2026-10-15 · open the original ↗

• The exhaustive deck answers the wrong question. The co-host, presenting, shows a 17-slide churn deck for a fictional B2B company (executive summary, methodology, cut after cut, key findings, recommendations) and says it answers "what data do we have?". An executive asking "how do the numbers look?" usually means "what should I do?". The co-host says the analysis underneath can be correct and still lose its impact because of where things sit.
• Lead with the insight: make each title the claim. The co-host replaces labels with findings: "Churn by segment" becomes churned accounts having skipped onboarding at a ratio of 4 to 1, "Q3 revenue overview" becomes "we lost $10 million in ARR and most of it was preventable", and "Key findings" becomes the decision to fund the onboarding push. The co-host then begins a structure of problem and stakes first, then insights backed by only the most salient evidence. The caption text stops there, after about 2,100 words of a roughly hour-long session, so the rest of the framework and the Claude Code demo are not in the source.

2026-10-15 · open the original ↗

• Without definitions, the model picks a column and sounds sure. The co-host, presenting, says an LLM knows what a customer is in general but not at your company, so it picks a column, a meaning for "week", and a view on whether a June dip is real, then answers confidently. Fixing this in the prompt works once and is gone next session, so the co-host says the prompt is never the place for it, pointing back to an earlier lesson in which asking AI the same question five times gave five different numbers.
• Onboard Claude like a new hire, on four things. These are what the data is (each file, what one row represents, what each column means), what the metrics mean, how the company reports (time windows, filters), and "what you would only know from the hallway", such as past incidents behind anomalies. In the co-host's example coffee subscription, "Brley", with two CSVs (metrics and ad spend), an activation means the day the first paid box is delivered, which is not the same as a signup.
• A definition you can't write plainly hasn't been agreed. The co-host offers this as a test for metric files: if you can't write it in plain, simple words, your team hasn't agreed on it: "a user who is active" defines nothing. The caption text stops here, after about 2,250 words of a roughly hour-long session, so the rest of the framework and the CLAUDE.md demo the title promises are not in the source.

2026-09-20 · open the original ↗

• The test asks whether the same question gets the same answer. The speaker opened four ChatGPT data-agent chats with memory disabled, attached the same synthetic grocery-delivery dataset (five tables, 12,000 customers, about 50,000 orders, no "active customer" column), and had Claude Code drive the browser through a fixed script of four turns: active customers for a Monday update, the trend, whether we are growing and what to do, and a shareable dashboard. The concern is scale: across a company of many askers, the agent may reproduce the familiar problem of the same metric name with different numbers.
• Four chats returned three different numbers, and none asked a question. Each agent noticed there was no active field and invented a definition: two used delivered orders in the last 30 days (2,194), one used non-cancelled orders (2,224), and one used 90 days (4,633). The speaker credits one chat for also showing the figure with cancelled orders included (2,268), but calls the failure to ask what "active" or "customer" meant an easy fail even for a junior analyst.
• Diverging methods cascade, even where definitions agree. On the trend, two chats said up 72% (from 1,275 in April 2025), one said 66.5% since June 2025, and one gave 3% over 30 days, 11% over 90 days and 44% year over year. The two chats that shared the 2,194 definition still disagreed on growth, 2.7% against the previous 30 days versus 0.3% week over week. All four reached the same conclusion, that the business is growing, with different numbers attached.
• Guardrails and run times also varied between identical chats. On the dashboard turn, two agents stopped and asked before uploading or sharing data, which the speaker praises, and two never asked; completion took 7, 9, 14 and 16 minutes, and one chat errored and was retried. The four finished dashboards each "look like the answer", and the speaker says none of their definitions came from the business.
• The speaker rates it a technical senior with junior judgment. The speaker calls the math and visualization excellent and 15 minutes for a dashboard "crazy fast", but says the agent is reliable only at guessing the definition on its own. The advice: ask which definition it will use before asking for a number, state yours if you have one, or request numbers under several common definitions and choose with stakeholders; the speaker argues OpenAI could fix much of this with simple early steps like those in the lab's own agentic-analytics systems, where the agent refines the question, agrees definitions and interviews the user first.

2026-08-26 · open the original ↗

• Cowork delegates the work; chat only answers turn by turn. Shane says chat can read uploads but cannot search on its own or save files, so its output "reads like a summary of the data" and the analyst ends up doing the analysis by reprompting. Cowork reads a folder and writes finished files back inside a sandbox, reaching warehouses such as Snowflake or BigQuery only through Anthropic-provided connectors. Claude Code runs on the actual machine with git, any database (even behind a VPN) and scripted, automated pipelines, which is what Shane would use to rebuild a nightly metrics job.
• An AI analyst is instructions, skills and connectors, not the model. In the lab's AI Analyst Plus plugin, the instructions are standing context, the skills are playbooks for one kind of work, and the plugin bundles skills and connectors. Each run loads instructions and a context store (metric definitions, dataset notes, past corrections), then frames the question as a decision, profiles the data, analyzes, validates, and only then delivers. Shane runs analysis on Opus 4.6 (described as released mid-February) and uses Fable or Opus 5 for building, saying later models cost more tokens and time and "go off the rails a bit more"; Cowork defaulted to Opus 5 at recording.
• A prompt should name deliverable, inputs, nuance and the decision. Shane calls the decision sentence the anchor: without it the analyst has to guess what the answer is for and returns something generic. The plugin includes gates that stop a vague request and interview the user instead. Shane also says Cowork tends to push Anthropic's built-in analysis skills and wander off answering questions nobody asked, so they add a global instruction and invoke the plugin with /analyst; in the demo it still pulled Anthropic's built-in data-viz skill.
• The demo returned a usable brief and flagged its own assumptions. On synthetic Novart e-commerce CSVs (orders, order items, products), with Opus 4.6 at high effort and a merchandising budget decision, it planned the work, then reported revenue up 41% from Q3 to Q4 2024 with every category growing, at between 36% and 51%. It recommended backing books and clothing and investigating electronics before any budget cut. It flagged that its revenue definition might be wrong because no context layer existed yet and that it excluded returns and cancellations. A chart rebuilt with the lab's data-viz skill stated the insight outright.
• Check AI output with no-code checks before sharing it. Shane's checks: trace the number the decision rests on back to its file and query, have another model check the work or run several in parallel to see if they agree, confirm category figures sum to the total (a mismatch points to a join that fanned out or filtered rows), and tie figures to numbers already known. Shane says the real answer is an eval suite and a shared context layer, because Cowork will not catch a metric definition changed in a meeting.

2026-08-13 · open the original ↗

• A written metric definition turns a capable agent into a reliable one. In the lab's test, a deliberately vague "what is our retention rate" was asked 30 times against a Snowflake warehouse, each in a fresh session, across Opus, Fable and the open model GLM 5.2. With no definition, 15 runs gave 8 distinct answers, each with valid SQL: Fable ranged from 9.2% to 25.3%, Opus sat at 34.9% (Shane calls it "the same wrong fork"), and the open model gave 99.2% on one run and 9.7% on another. With the definition in a semantic layer, 15 of 15 runs returned 25.6%. Shane advises writing meaning, grain, numerator, denominator, window and filters with no SQL, because SQL rots as tables change and the meaning does not.
• Declaring joins and cardinality stops silent row duplication. A correct definition says what to count, not which path to count it along. In the lab's synthetic data the same measure came back as 5.9 million or 3.1 million depending on how the agent walked the joins. Shane says a compiled semantic layer (dbt's compiler refuses a duplicating join) is more deterministic but refuses questions off its menu, while plain text-to-SQL with joins and cardinality described keeps that range.
• Verified worked examples moved accuracy most; unstructured context lowered it. Shane cites a LinkedIn ablation test over 130 questions, scored as the share of answers an expert rated 4 out of 5. Adding table structure scored 9%, groupings 11%, descriptions 24%, verified queries 49%, and crowdsourced domain knowledge and jargon dropped it to 42%. Shane calls verified queries the heaviest lift because query history contains wrong SQL, and says to use a given pair either to steer at runtime or as a held-out answer key, never both.
• Gate only where the metric would fork on a guess. Shane says agents spot ambiguity well when asked but otherwise run with the most reasonable reading, so the lab adds either a sensible default (Looker's last 12 months when no date range is given) or a skill that marks the question unclear and stops. Gating too broadly, or asking bad clarifying questions, adds friction and pushes the work back onto the human.
• Unstructured documents should feed structured context, not runtime. Shane says docs and Slack add conflicting, half-correct readings, so they should be used to draft candidate definitions that a team aligns on, and the agent then reads only the structured result, which also saves tokens. Vendors now draft context with agents but keep a human approval step. Shane predicts that within a year or two a named context-management function will emerge between data teams and stakeholders.

2026-08-13 · open the original ↗

• Open weights mean owning the model rather than renting it. The presenter gives three reasons that matters: data never leaves your own hardware, the model can be fine-tuned on proprietary data, and nobody can withdraw it. As an example of withdrawal, the presenter says Anthropic's Fable 5 was pulled a couple of days after release when the US government called it too risky pending review.
• Open frontier models now trail closed ones mainly on price. On Moonshot AI's own benchmarks for Kimi K3, only Fable 5 and GPT 5.6 clearly lead, a gap the presenter says still existed around May. Fable 5 costs $50 per million output tokens, which the presenter puts at orders of magnitude above Kimi K3, GLM 5.2 and DeepSeek V4 Pro. Citing OpenRouter, the presenter says open models took about 20% of routed tokens but 4% of revenue in 2025, and crossed half of routed tokens by mid-2026. The presenter advises taking benchmarks "with a grain of salt" and testing on one's own workflows.
• Frontier-scale open models need hosted or self-hosted servers, not laptops. The presenter lists three ways to run one: locally with tools like Ollama, through a hosted API, or self-hosted on private servers. A standard 16 GB Mac mini (about $600) runs roughly 8 to 16 billion parameters, against 2.8 trillion for Kimi K3, so local models suit simpler tasks such as summarizing or copy-editing.
• Kimi K3 inside Claude Code found the planted Simpson's paradox cheaply. An environment script pointed Claude Code at Kimi's API to run the lab's AI Analyst Plus repo on the fictional Novart data, asking how checkout conversion had trended and how to optimize it. An earlier run's report found completion falling from 68% to 30% while traffic grew 27x, attributed it to mix shift rather than a broken checkout, and named mobile checkout-to-payment-start as the fixable gap. The presenter calls these the known correct answers. That run cost about $5.60 by the presenter's receipt script, or roughly $3 by the presenter's own usage counter, against about $16 at list API price for Opus 5, which gave a similar result. The live run was not reviewed.
• Open models fit high volume, control, and cross-checking other models. The presenter says there is a volume break-even past which paying for closed models stops making sense, and that model families can check each other's work. Shane adds a personal view: analytics capability "hit that threshold" with Opus 4.6 in February, so newer frontier models don't necessarily do the job better. Asked about fine-tuning, the hosts say the lab has never needed it, and see it as suited to large proprietary datasets, healthcare, or trade secrets.

2026-07-30 · open the original ↗

• Choose the Claude layer by how far you need to go. The speaker ranks three layers: chat for a quick check on a pasted CSV, which forgets the file in a new chat; Claude Cowork, which works on real files and folders with no terminal or setup and asks follow-up questions before guessing; and Claude Code, which runs analysis end to end in the terminal against the data warehouse and is the most flexible but has a steeper learning curve. Their rule is "chat for something fast, Cowork to actually build, code to scale", with Cowork recommended as the gentler way in. The clip promotes a free live session on August 26 that builds a first AI data analyst in Cowork without code.

2026-07-25 · open the original ↗

• Measure-and-share analysis rarely drives a decision. The speaker says most naive analyses run two steps, measure and share. The lesson teaches six (decide, ask, hypothesize, measure, share, act) and later adds an explicit validate step to make seven. The running example is an e-commerce "save for later" button: savers buy more, so should the button be more prominent? The number may be correct and still not support a decision.
• Pin down the decision's stakes and the question's definitions first. Deciding includes the cost of the work the team won't do instead, and someone has to own the call. On "buy more", a room the speaker polled gave revenue per session, revenue per customer, orders per month, AOV per user, repeat orders, subscriptions and referrals. Five analysts, models or AI runs could return five technically correct answers. The session defines a buyer as a user with at least one order of any status, ever.
• Bare Claude Code returns a convincing correlation it cannot check. With one three-column file (user ID, saver flag, ever-purchased flag) and no skills or agents, it reported 68.6% of savers vs 52.6% of non-savers had ordered: a 16-point gap, about 1.3x or 30% more. It flagged on its own that this is a correlation, not an effect, but had no data to test that. The speaker argues the friction AI removed used to force the "is this worth asking?" step, so validation now has to be made explicit.
• Write the disproof and the funding threshold before looking. The hypothesis is that saving causes some buying; it would be disproven if the same people buy more for another reason, such as tenure. The bar was set at an expected 5-point lift, because a bar set afterwards gets talked into whatever turns up. In a gated "workbook" repo with a skill, recorded definitions and added columns (signup date, channel, country, device), the system flagged tenure (700 vs 616 days on average), refused post-saving variables, and stopped at gates for the user's confirmation. Within tenure buckets the gap fell to about 5–7 points, so the speaker would probably not fund the change. They note this is still not causal and that a randomized experiment would settle it.
• AI leads only the measuring; the human owns everything else. In the speaker's two-lane chart, measurement is the one AI-led stage. AI can draft the rest and run validation checks, but those are checks the analyst wrote in advance, and the analyst is accountable for the number just as with a spreadsheet or Snowflake. Their single takeaway is to ask "what decision is this for?" before typing anything.

2026-07-20 · open the original ↗

• The lab argues open-source models are where AI analytics is headed. The speaker announces a partnership between the lab and Moonshot AI, maker of Kimi, and says Kimi K3, released "last week", is the largest open model ever, ahead of GPT 5.5, Opus and other open models on published benchmarks, ahead of every model tested including Fable on some agentic work such as long-horizon coding, and about half the cost of frontier closed models. These benchmark and cost figures are the speaker's claims as of recording. The pitch is for a free session, lesson four of the 15-lesson "Build Your AI Analyst" series, that tests Kimi and other open models head to head against Claude on quality, cost and speed.

2026-07-20 · open the original ↗

• Most analysis dies by being ignored, not by being wrong. The speaker names three ways it happens: pulling data before knowing which decision it serves, measuring everything (47 metrics, a dashboard for every team, refreshed daily, none forcing a decision), and "let's take this offline" with no owner, date or decision. Their fix is three questions before starting: what result would make us act, who acts, and when. If those can't be answered, do no analysis yet. They argue this matters more now because AI has made getting the answer the easy part, which leaves asking the right question and landing the decision as the scarce work.

2026-07-16 · open the original ↗

• Claude answers a bad question as confidently as a good one. The speaker gives this as the reason for the lab's free "Build Your AI Analyst" series: 16 weekly live sessions from July through October. It opens with Product Analytics 101, which starts from the decision to be made rather than the dashboard at hand. The announced topics include full analyses on open-source models, pitched as cheaper and more secure, semantic layers so that the same question gets the same answer every time, and AI data science and analytics interviews.

2026-07-15 · open the original ↗

• More output pays only if a decision loop surrounds it. The speaker's loop is to name the decision (what call, who makes it; if nothing changes either way, skip the work), then sharpen the vague ask into an answerable question. When Claude hands over a convincing number, ask what would have to be true for it to be wrong and check that one thing. Then go back a month later to see whether the number moved as predicted. They note that a good question alone is not enough, because Claude answers a bad one just as well.

2026-07-09 · open the original ↗

• Skills, agents and hooks turn a confident intern into a reliable worker. The presenter treats Claude Code as a new hire. A skill is the playbook, a plain-English file defining how a task is done; an agent is a teammate you delegate to, with its own persona and context; a hook is a rule enforced deterministically at a trigger point. Answering an attendee, the presenter calls the whole package in a folder (skills, agents, hooks, helpers, instructions) the "harness".
• Use hooks for what must happen, not for everything. The host adds that a skill's main failure mode is not firing, especially in a large system or with an overgrown CLAUDE.md, so a must-run task belongs in a hook. Hooks lose the model's judgment about when to act, though, so not everything should be one. The host's own use ties hooks to evals through traceability: every Snowflake query call triggers a hook that logs the SQL, a timestamp, an ID and the question's context.
• A skill and a critic agent produced and stress-tested the weekly report. In Claude Code Desktop on a fictional coffee-subscription app's data, /weekly-review followed a skill requiring a TL;DR, the one number that moved, one risk, one action, under 250 words, numbers only, and percentages to one decimal. The report traced a double-digit week-over-week drop to two days when paid social reported almost no activity. After the presenter edited the skill to open with a status light, the rerun rated it yellow. An "exec reviewer" agent playing a VP of growth then returned three likely questions, including whether the signups were "lost or just miscounted" and why organic peaked on those same days.
• A data-integrity hook refused to ship a report on broken data. The presenter changed one value from 772 to 900 so that activations exceeded signups. A Python hook checking for duplicate date-channel rows, activations above signups, and negative or malformed values then blocked the report from being written, with no prompt telling it to run. The presenter also points to the lab's open-source "AI analyst" repo as the fuller version of this harness, taking a business question to a deck or one-pager.
• Every analysis is a bet, and AI improves all three terms. The presenter's formula is impact = number of bets × hit rate × size of the win. On their account, the system gives more shots (the Monday hour becomes about five minutes), a higher hit rate through deeper slicing, and room for "venture bets" that were once too costly to take.

2026-07-04 · open the original ↗

• Check the question before touching the spreadsheet. The presenter's four-step loop is ask, check the data, visualize, summarize. Skipping the first step makes everything after it meaningless. "Analyze the spreadsheet" or "is the product healthy?" are bad asks; "is paid search still profitable this quarter?" names the product, the measure and the timeframe.
• Run three tests before trusting any number. The tests: totals (segments, if mutually exclusive and collectively exhaustive, should sum to the top line), magnitude (is it in a plausible range, for example within a standard deviation of its normal variability allowing for seasonality), and one known number verified against a value you already have. The presenter says these translate directly into evals. They describe a metric anomaly engine at a previous job that ran these checks weekly before the metric's owner saw the result.
• Gemini in Sheets handled simple asks but dropped a requested view. On the lab's synthetic e-commerce data it charted the three-month revenue trend, added a day-of-week column with a visible formula, and ranked months by revenue (December highest, January lowest), consistent with the chart. Asked for a tracker with week-over-week and month-over-month views, its execution plan left out month-over-month, so only week-over-week was built. The presenter thinks a clearer prompt might fix that.
• Claude Code built a full tracker fast because the skill already existed. Running Opus 4.8 with 1M context in auto mode, it built an October 2024 Google Sheet with top-line metrics, breakdowns by channel, device and category, week-over-week changes, conditional formatting, a revenue chart, and an insights tab comparing against September. The presenter says it took about a minute, under 5 minutes in all, because a create tracker skill, a Python orchestrator and a Sheets API wrapper were already in the repo. The first build took about 10 minutes of iteration, and their Google Sheet creator agent was not called this time. They concede the comparison with Gemini "cheated", since Gemini had only a prompt.
• A sheet lets non-SQL readers trace AI-generated numbers back. The host argues that shipping a sheet alongside a doc or deck gives stakeholders visibility to validate an agentic analysis without reading logs or SQL. The presenter adds that a pipeline could log each step's query and data to its own tab. Asked about dynamic scenario models, the presenter says they have not done it in Sheets but have built parameter-driven HTML dashboards, with a Python file the skill triggers on matching inputs.

2026-06-25 · open the original ↗

• A wrong AI analysis looks exactly like a right one. Shane says it produces a plausible number and a convincing story either way, and the cost is broken trust if someone catches it or a bad decision if no one does. He recaps last week's reliability eval: the same retention question to five separate agents returned five defensible answers from 30% to 99%, and writing down the definition made runs agree.
• First ask whether an answer key exists. Where one does, start with "found truth" — finance-reconciled figures, board metrics, queries verified many times — and run it like unit tests; for questions that drive decisions and have no key, "forge truth" by working out the answer once and saving it, which costs hours to weeks. He warns that a metric can be quoted wrongly for years, and an eval graded against old buggy queries teaches the AI the same bugs.
• Without a key, grade the work, not the answer. His four checks are reliability (does it agree with itself), receipts (the exact SQL, tables and metric), reconciliation (segments sum to a trusted total), and reading the trace for a defensible method. None proves the answer; each "shrinks the space where a mistake can hide", and each has a blind spot: a stable wrong answer, an error that runs through every segment, an LLM-written receipt that is itself a hallucination (so log with hooks and generate receipts from code), and a definition error that reads fine.
• Every eval ends in act, dig or abstain. Shane frames abstaining as a real decision that builds stakeholder trust when confidence is low, and says rigor should match the stakes — deeper for the board, lighter for a curious stakeholder, with the cheap checks first where data is messy. A caught error becomes context for the system, but should not be trusted until verified, so it does not poison later answers.
• The demo's audit caught a join that inflated revenue. Asked whether a revenue-by-category query on the fictitious Novamart data could be trusted, Claude Code (using CLAUDE.md risk lists and a semantic layer) found a join fan-out — 47,000 orders exploding to 75,000 line-item rows with an order-level total summed — giving 6.9 million against about 3.6 million, an 87% overstatement. It also flagged no filter on cancelled or returned orders (a further 15%), a total that includes shipping and should be line total, and that the data stops at 2024; Shane pasted the query by hand here but says in practice a hook records every query to JSON.

2026-06-18 · open the original ↗

• Trust in an AI analysis is a dial, not a switch. Shane defines an eval as a check on how far to lean on an answer you did not work out yourself, with rigor matched to the stakes: a quick look for a curious question, real checking for a number going to the board or the street. Because analysis is a pipeline of at least seven steps (interpret, pick tables, pick metric definition, write SQL, run, read, recommend), he says early errors such as a wrong active-user definition carry cleanly into every chart after them, so that is where to spend effort.
• Four checks cover almost any output. They are a known answer (ground truth), a rubric (refined by humans first, then scaled with an LLM as judge), asking several ways, and showing the receipts — the exact SQL behind each number, which he calls the cheapest and says analysts should do regardless of AI. His caveat: agreement across runs or models is reassurance, not proof, since a system wrong for the same reason agrees with itself, and he finds disagreement the more useful signal because it says where to dig.
• A stable answer can hide an unchosen definition. In the demo, a reliability skill in Claude Code ran "how are we doing on checkout conversion?" through five independent subagents against Snowflake and got 33.2% every time. Shane says the stability came from a metric dictionary pinning one of several defensible readings (checkout start to payment, all sessions, cart to checkout, people versus sessions), so the check showed consistency, not that the definition was the one stakeholders meant.
• An undefined metric produced five different retention numbers. Asking "what's our retention rate?" with no dictionary entry returned 99.3%, 99.2%, 9.7%, 34.9% and 78%, each a real definition (month-after-signup activity, 30-day repeat purchase, two-plus orders over a year, a second session within seven days). The check flags the problem but cannot pick the answer; the fix he gives is agreeing a definition with the team and storing it as context, such as a YAML metrics file, and a hook logging every Snowflake query supplies receipts that the model cannot hallucinate after its context is cleared.
• Better models will need more checking, not less. Shane predicts models will get better at being right and at being "super convincingly wrong". In Q&A he says that inside a built harness Opus 4.6 matched or beat Opus 4.7 and 4.8 and was on par with Fable, which won only with no system around it — his basis for expecting open-source models to catch up; for security-restricted firms he cites anonymized data and hosted, zero-retention setups, and a co-host adds locally hosted open-source models.

2026-06-16 · open the original ↗

• Most teams measure activity instead of customer value. Shane says logins, sign-ups and revenue go up and to the right without showing users got value, which shows up as a vanity metric or a dashboard of 30–50 numbers no one rallies around. Because the North Star sits above every decision, a wrong one passes its mistake to everything beneath it; he cites a former employer whose weekly active users counted anyone who opened an email, which iOS auto-opens inflated.
• Four of seven North Star tests are deal breakers. The seven are customer value, fit with strategy, leading rather than lagging, movable by the team, understandable, measurable, and free of vanity; failing customer value, leading, movable or not-vanity disqualifies a metric regardless of the rest. Airbnb's nights booked is his passing example; the frameworks are drawn largely from Amplitude's North Star playbook, and the lab says it is partnering with Amplitude on this session.
• Input metrics must decompose the North Star, not rename it. The levers are breadth, depth, efficiency and frequency (how many customers, how much each does, how reliably it goes through, how often they return). Shane warns that "active buyers" under a "weekly active buyers" North Star is the same number in a different hat, so pick a countable event such as orders or nights booked that splits into separate levers.
• The demo audit refused DAU and graded completed orders weak. A North Star skill in the lab's course-only "AI analyst plus" repo, not yet pushed to main, grounds each verdict in a cited knowledge base via a "librarian" agent. It refused daily active users as a canonical anti-pattern (logins, not value received) and suggested a value-delivering action instead; weekly completed orders passed five of seven checks with no fatal failure but was flagged "issue-blind" to returns, defects and late delivery, with the fix "weekly completed orders with no issues", plus a generic strategy fit that more business context would sharpen.
• A strong headline number hid growth built on one lever. The drivers mode on the synthetic Novamart data for 2024 (served from an earlier cached run) found orders grew six times, with about 99% from breadth, flat frequency, average order value down about 9% and the membership program at 2–3%. Shane reads this as "renting new customers" and says the AI can grade and decompose a metric but the choice of what to measure and which lever to prioritize stays a human decision.

2026-06-16 · open the original ↗

• Packaging costs more than analysis because its steps run in sequence. The presenter says formatting a finished analysis into a document — charts, titles, placement, coherence between sections — often takes longer than the 30–60 minutes of thinking, and every analyst on a team pays that tax each time. The proposed fix is to analyze once and repackage per audience, with formats and templates decided in advance so Claude Code can publish in minutes; the presenter concedes that scaffolding takes time to build.
• Every publish prompt should name four things. They are the audience, the format, the length and depth, and what to include. The demo's map gives each format a different job: a Google Doc for a VP as a decision record (executive summary, findings, recommendation, the decision needed), a Notion page as a detailed team wiki carrying all charts plus SQL and methodology for review, and slides for a 5–10 minute slot with one message per slide, a headline that states the point and a chart as the proof.
• The hard part is choosing what to cut. The presenter says junior analysts put everything in, leaving 50–60-slide decks to cut to 15; the host adds a new graduate whose 200-chart deck came down to seven. The generated documents titled every chart and section with its insight rather than a description, which the presenter singles out as the point.
• The demo published one finished analysis three ways. Working from the synthetic Novamart dataset (13 tables, 6 million records, a local DuckDB), Claude Code summarized 2024 as revenue up 5.4x to 3.15 million with growth "leaky and getting more expensive", then wrote the VP Google Doc through the Google CLI, a multi-tab Notion page through an authenticated MCP (shown from a run made before the session), and a first-draft slide deck. The presenter says the analysis was already done, which is why it was fast, would still iterate on the Notion layout and on the slides' thin text, and skipped Slack and email.
• Audience feedback should flow back into the next version. Answering an attendee, the presenter describes feeding an Otter transcript via MCP to Claude Code after each meeting to plan a V2, and having it read and answer leadership comments in Notion or Coda. The host adds that he analyzes Claude Code workshop chats and transcripts the same way and built profiles of his main stakeholders from their documents and Slack messages to tailor what he presents.

2026-06-16 · open the original ↗

• A/B tests go wrong because the reader wants a result. Shane says bad tests come less from weak tooling or rigor than from the person reading the result being the one who hoped for it. He names four habits: moving the threshold to where the result landed (a 2% minimum detectable effect quietly becoming 1.5% after a 1.6% lift), peeking and stopping while winning, explaining away broken guardrails, and promoting a secondary metric when the primary is flat. He calls this how brains work, not dishonesty.
• Let code decide, LLMs translate, and humans judge. Experimentation platforms compute the statistics correctly but, in Shane's account, act as calculators that leave the call to people at the moment their judgment is most compromised, and LLMs pattern-match toward hoped-for results the same way. His system targets design and analysis and deliberately leaves instrumentation alone as too high-risk. Deterministic Python issues the verdict, LLMs draft and explain, and judging agents get less context on purpose — "blind judges" — because the failure being defended against is motivated reasoning, not ignorance.
• Metric definitions live in files the agents cannot invent around. The open-source Agent XP repo has five agents (understander, designer, critic, SQL, narrator), design and analyze skills, and a whitelist of stats functions the critic checks were used. Project-level YAML files define semantic models and the demo's 14 metrics, each carrying its type and defaults; revenue per user, for example, brings a −1% guardrail to every experiment that uses it. Shane says deciding those definitions remains a human conversation with the team.
• Analysis runs as ordered gates before any lift is shown. The steps check that the split matches the brief (a 57/43 split on a planned 50/50 halts before lift is computed), that no pre-set guardrail broke, and that the sample is large enough. Only then does it read the primary lift's direction and magnitude, then whether the effect held over the window (for example a 25–30% drop in the last third flags novelty), before a ship, don't-ship or learn verdict.
• The live demo produced a sealed design but never reached analysis. The run took 13 minutes. For "move the buy now button above the fold", the designer picked conversion rate as primary from the catalog because it matched the word used, proposed buy-now click rate as secondary, and ran power analysis on the primary and on the guardrails, finding a 5% relative MDE feasible. A critic agent then reviewed it and the brief was sealed. The analyze stage was not run, and Shane says the tool does not do randomization or assignment and is early, actively changing work.

2026-06-16 · open the original ↗

• Pushing back on the question is the highest-value analyst skill. The presenter says students in the lab's first five-week cohort named reframing the question as the single most valuable thing they learned, ahead of the repo of 60 skills and agents. A vague ask like "analyze user engagement" yields generic observations no one can act on; "which segment drove the engagement drop last month so we can decide where to focus" names a segment, a quantified driver and the decision.
• Every question must pass three checks before analysis starts. The presenter's 30-second gut check asks whether the question is tied to a decision (if none can be named, stop), grounded in data that actually exists, and specific enough to know when you are done. The worked ladder runs from "why are signups down?" to "did the pricing page redesign cause the 12% signup drop in the 2 weeks post launch?", with the decision being whether to roll it back.
• Interview prompts are vague on purpose and reward clarifying first. Asked how to measure a new search feature, candidates who list DAU, retention and engagement fall into "the trap", the presenter says, while those who get offers ask success for whom (power users or everyone) and against what baseline. Only then do they set a primary metric, a guardrail and a 30-day bar for continued investment, confirming the decision with the interviewer as they go.
• State in advance what result would prove you wrong. For a hypothesis that mid-market churn comes from failed onboarding rather than pricing, the presenter lists confirming evidence (churn clustered in accounts that never finished setup) and rejecting evidence (churn spread evenly, pricing dominating exit interviews). Requests are then scored on impact and feasibility: do high-impact, data-ready ones first, plan ones that lack instrumentation, use quick wins as buffer, and decline the low-low "graveyard of side quests" with written reasons.
• The demos turned a churn Slack into a bounded question. A "question quality coach" prompt in the Claude web app (work, interview and rank modes) failed "can you look into our churn numbers?" on two of three checks and coached it toward a segment-ranking decision. A question-framing gate in the lab's Claude Code repo scored it 0/3 and, filling gaps with assumptions for the demo, produced: has monthly voluntary logo churn risen above a 3% target over two quarters, in which plan-tier and tenure segment, so leadership can decide on funding a retention intervention — with "below 3% and no single segment" as the stop condition. The host adds that vagueness often hides in windows, filters and thresholds.

2026-05-21 · open the original ↗

• Define and size the move before slicing anything. The presenter, a co-host, calls this "level zero": state the metric's definition, grain and window, say whether the change is absolute or relative and against what (prior four-week average, week-over-week, year-over-year), and set a bar for whether it is worth chasing. A 12% drop that follows an unexplained 12% rise is just a return to baseline.
• Drill into the dimension that explains the most change. The biggest percentage move is not the target. In the presenter's example, revenue is down 12% ($5 million) and APAC accounts for $4.5 million of it. That is the direction to follow, even if one acquisition channel is down 80%. Slice again inside that segment while it still carries most of the top-line move, from "EMEA" down to paid members in Germany or the UK on iOS. The failures named are panic, telling a story from the one dimension a dashboard has, and "drilling to nowhere", which Shane adds turns up spurious correlations and red herrings.
• Agents add depth a human would skip, not just speed. Shane recalls spending two months at Nextdoor on a root cause analysis of falling sessions, losing track of threads as new data came in. He argues that subagents can drill dimensions in parallel and write to one knowledge base, and that even naive segmentation and attribution done this way is "extremely powerful", with causal-inference methods as an optional layer on top.
• The demo traced a June 2024 ticket spike to one iOS release. On the repo's fictional Novamart data (50,000 users, 47,000 orders, 6.5 million events, 500 products, a $99 membership), a plain-English prompt had Claude Code find that payment tickets jumped to 41% of all tickets in June and fell back in July while attempts grew smoothly. It named a defective iOS app release, version 2.3.0, fixed a few days later, and rated its own confidence high. The presenter says the work still has to be checked. The Google Doc shown was produced beforehand, and the presenter says they already knew the story the data held.
• Opus is required for reasoning work, by the hosts' account. Shane says the system "does not work well with Sonnet", which is "very confidently wrong". It only clicked with Opus 4.6 in February, and Opus 4.7 went "off the rails" until the paid version of the repo was updated. A third co-founder says Sonnet can do pure execution of fixed skill rules. In Q&A the presenter says the agent flags every cause it finds when several drive a move, not only the largest. It connects to Snowflake and DuckDB natively, has no Copilot version, and a Codex version is being built.

2026-05-16 · open the original ↗

• Every success metric has a shadow that worsens under optimization. The presenter, a co-host, opens with a checkout feature that lifts conversion 15% while net revenue falls two weeks later. The pairs given are conversion rate against average order value, sign-ups against day-30 retention, engagement against NPS or complaints, and support speed against tickets reopened. The presenter argues that learning how metrics move against each other matters more than any single launch decision.
• Fix three things before launch: metric, guardrails, thresholds. The success metric must be specific ("purchase rate for new users", not "engagement"). Guardrails are named in advance rather than "track everything". Hunting for supporting metrics after launch is "data theater". Each guardrail gets an alert threshold (a 3–5% move might be tolerable) and an absolute kill threshold that pauses the feature.
• Pre-set thresholds keep the decision out of politics. Shane argues that fixing numbers in a document before any data exists keeps finance, sales and product teams from reading results through their own priorities. Without that, he says, post-launch conversations turn into conflict and get political. The presenter adds that the more sensitive the experiment, the more this matters.
• The demo designed guardrails for a "save for later" launch. On the fictional Novamart data, with a 30-day purchase rate target of +15%, Claude Code on Opus 4.7 flagged cannibalization of immediate purchases as the main risk. It proposed same-session purchase rate, average order value, return rate and 90-day repeat purchase rate. Told return rate isn't tracked, it substituted support-ticket rate and pre-shipment cancellation rate. For returns rising while 90-day purchases stayed flat, it read the feature as changing what customers buy, not whether they return. The presenter says the acceptable ranges came from earlier runs on the same data and can be set explicitly.
• Have a fresh-context subagent check the work, not the builder. Answering an attendee, Shane says the agent that did the work is biased toward judging it correct. He uses subagents with clean context and a skill in which several personas critique separately, argue, and converge. He also uses a second frontier model such as Codex. On data, he says to feed raw numbers, because the LLM writes Python to compute and never ingests them. Numeric work should stay in code to avoid hallucination, though rounding may help if the model pattern-matches on numbers directly.

2026-05-14 · open the original ↗

• A sizing must separate assumptions from math. The presenter, a co-host, says most sizings take an optimistic assumption, multiply it by a large addressable market and hope nobody asks. The alternative is one formula with three inputs: users affected × improvement rate × value per unit. Each input is then marked as grounded in data or assumed, so the assumptions can be debated.
• Each input needs a traceable number, not an adjective. The presenter's examples are 12,400 monthly mobile users who start checkout and don't finish, closing half the gap between 33% mobile and 52% desktop conversion to reach 42%, and a $67 average order at 75% margin, about $50 per conversion. A vague $5 million "save for later" pitch is rewritten as 28,000 monthly users who view three or more products without adding to cart, 8–12% recovery from an industry benchmark, and a $45 first purchase. The presenter argues that the first traceable version sets the anchor the audience holds on to.
• A role prompt can grade a pitch input by input. A chat prompt casts the model as a senior product analyst and has it rate each input as clear, vague or missing, then rewrite the pitch with numbers. Given only "improve mobile checkout, it's clunky and we're losing sales", Claude marked all three inputs as weak and plugged in an assumed 500k users for a midsize site. A director in the audience objected that a finance persona and an analyst persona should give the same number. A third co-founder replied that the number should match when it is deterministic and only the framing differs, and called team-wide consistency a validation problem.
• The demo sized mobile checkout at roughly $300k in topline revenue. Opus 4.6 was no longer selectable, so the presenter used Opus 4.7, which the presenter says was released the day before. Claude Code found web converting at 40% from start to complete against 29% on iOS and 28% on Android, and its Simpson's-paradox check found web ahead in every channel. The repo's opportunity-sizer agent took a 15% relative mobile lift as the base case, with 5% as pessimistic, and gave a sensitivity table of lift against order value. It rated confidence medium: reach, order value and margin were data-backed, the lift was not. The presenter says it had no business context and its assumptions are for a human to challenge.
• Agents need context beyond the warehouse. Answering a question about connecting BI tools, the presenter says Claude knows nothing of a company's strategy, roadmap or past analyses out of the box. Notion pages, Google Docs, email and Slack, anything with an MCP connector, count as context and improve its answers, subject to the organization's security and legal approval. The presenter adds that the repo's skills were built by encoding the founders' practices and checking where Claude's defaults fell short, and that each user's copy grows to fit their own company.

2026-05-14 · open the original ↗

• AI decks look generic because the prompt gives no instructions. The presenter, a co-host, shows "pretty mid" slides from a bare Claude Cowork prompt. The argument is that the model takes the fastest route to the goal unless told otherwise. Four inputs fix that: the audience (a CEO deck hides 90% of the method a data-science deck would show), the decision the deck drives (ship, on track, or a resource ask), a storyboard before slides (TL;DR, context, findings, recommendation, never "a chart dump in chronological order"), and a theme.
• Build decks in a pipeline with automated review loops. The repo's sequence runs storyboard, narrative, charts, compiled deck, then human iteration, and each stage loops back until reviewers in the system pass it. Shane says the first draft then arrives already through what would be four or five rounds of review by colleagues and stakeholders, with passing criteria the user sets. The presenter says the last 10% of the work drives 90% of the impact.
• The demo built a New York Times–themed executive deck unprompted on design. On Opus 4.7, Claude Code listed six themes, including corporate, minimal, New York Times and Economist. It then read a finished mobile-checkout analysis on the fictional Novamart data and wrote the theme's CSS, rebuilt the charts with takeaway titles, and rendered a PDF and then HTML. The result was a short deck ending in an ask, with an appendix, built without any display instructions. The presenter thought it should have been wider. Shane says themes Claude invents fresh each time "can be pretty bad", so the repo predefines them with slide types, though Claude still creates new layouts.
• Fix it once, then codify the fix into the skills. Shane says Claude will make one-off fixes to the deck in hand by default. After each deck he asks what changed between draft and final, and updates the templates, skills or agents so the next deck starts aligned. He also says Claude Code can trigger deck-building automatically at the end of an analysis that starts from a stakeholder's Slack question.
• Claude Design is currently weaker than Claude Code for decks. Asked by an attendee, the presenter says Claude Design, released about three weeks earlier, is likely to improve. But two decks used up all the presenter's credits, and instruction-following and grasp of context were "pretty rough". Claude Code is more flexible, its cost more predictable, and it chains into other workflows. Other answers: output can vary between runs and between Claude Code and the repo's Cowork plugin, the $20 Pro plan is enough, and since Claude Code runs on Anthropic's servers a separate machine matters mainly for fully autonomous agents.

2026-04-16 · open the original ↗

• Interviewers score structured thinking, not knowledge. The presenter, who says they have run hundreds of hiring-manager interviews, names five dimensions (scope, metric, decomposition, trade-offs, decision). Strong candidates mostly fail by jumping to answers, asking clarifying questions only for form, and never ending on a recommendation. The presenter says this rubric gets stricter the more senior the role.
• A good answer clarifies, decomposes, offers branches, then decides. The worked case is a free trial launched 3 months ago with signups 40% above target and revenue 50% below. The presenter's answer asks first whether that means total or per-trial revenue and whether the lift was organic or paid. It then splits revenue into trial starts × conversion × ARPU and tests three hypotheses: low-intent users (check signup source mix), a trial that doesn't show value (feature adoption in the first 7 days) and pricing friction (checkout abandonment). The candidate asks the interviewer which branch to follow, weighs cannibalization and ROI, and closes with a decision for the VP.
• Classify the question type before structuring the answer. The presenter gives five types with DoorDash examples: measuring success (DashPass), diagnosis (orders down 15% last week), experiment design (a checkout A/B test), launch-or-not (rural expansion) and improvement (dasher retention). One interview can chain several of them, e.g. metrics, then test design, then reading fake results, then a launch call. The presenter says PM analytics interviews are similar, while data science goes deeper into statistics and causal inference.
• A staged agent pipeline in Claude Code built the prep pack. For a staff data scientist role at DoorDash, a company intel builder skill calls three agents: an earnings researcher, a blog synthesizer and a competitive analyst. A metric tree builder and an experimentation researcher then run in parallel, followed by cheat sheets and an HTML stage, ending in 12 markdown files and a 7-day plan. The GOV metric tree went from orders × AOV down to new, retained and churned consumers, and DashPass versus non-DashPass users. The presenter says the pipeline works best for companies that publish heavily, as DoorDash does on its data and ML blogs. Asked about hallucination, a co-host said they believe the pipeline ends in a fact-checker step; the demo did not show one.
• A mock-interview copilot runs in simulate, coach, practice and full-answer modes. It is a Claude project loaded with a smaller generic analytics document set, because the presenter could not load all 15 files at full depth under Claude Pro. It was not given the DoorDash material. In simulate mode Claude played both a fictional Meta interviewer ("Priya") and the candidate, in chunks, and the candidate opened with clarifying questions. The demo did not get as far as scoring an answer.

2026-04-04 · open the original ↗

• Designing the wrong analysis wastes more time than bad SQL. The presenter's test case is a VP saying "conversion dropped last week" and wanting an answer within an hour. The first step is to clarify which conversion is meant, compared with what (week over week, year over year), and why the VP is asking. Given only that one sentence plus the repo's CLAUDE.md and skills, Claude Code asked those questions, ranked seven hypotheses by likelihood, named three checks to run first and drafted a Slack reply. The presenter notes that the ranking reflects generic patterns until it is given company context.
• Fill a seven-line design brief before running anything. The lines are question, decision, hypothesis, comparison, segments, confounds and criteria, and the presenter calls the brief "a thinking tool", not a checklist. The decision line means asking the stakeholder what they would do with the answer: "just curious" is not a decision, while "rework checkout or brief marketing, by a deadline" is. A hypothesis must be falsifiable, naming the magnitude, the category and the customer type, because "pull the data and confirm" builds in confirmation bias.
• Comparisons, segments, confounds and criteria are where analyses fail. The presenter's examples: prefer a natural experiment (mobile removed a widget, desktop kept it) to a default week-over-week comparison that a holiday or promotion can distort. Check segments specific to your company for Simpson's paradox; in the presenter's company, whether a user had previously paid is one. List concurrent changes, such as a loyalty restructure, a 15%-off promo or a changed mobile tracking pixel. Write down the accept and reject criteria first, e.g. accept if mobile consumables fell more than 3 percentage points and desktop did not, reject if the drop is uniform. The presenter's slogan: "2 minutes of design prevents two weeks of wasted work."
• An analysis-design skill turned a PM's hunch into a V1 plan. The demo is "CartLoop", a home-goods marketplace whose PM thinks removing a post-purchase recommendations widget cut repeat purchases. A hypothesis sharpener drew out the assumptions (the widget is the main driver, the drop is uniform, nothing else changed) and set comparison groups and accept/reject criteria. A confound scanner examined the claimed 3.77 pp decline for product, marketing, technical and data-quality changes and selection bias. A third stage ranked the investigation. The plan was exported to a Google Doc after an expired auth token was renewed live. Running V1 on the data and the V2 plan revised after stakeholder feedback were not done live: the presenter showed copies made beforehand.
• The host predicts analysts will build agent systems, not analyses. He says the job is shifting toward building agentic systems that do analysis, which for now still need heavy validation. He expects that need to shrink, the way hallucination checking did after GPT-3.5. He says he feeds Claude Code context through Notion, Google and Slack MCPs, repos of past analyses and the product codebase. He also says he never wants to maintain a dashboard again and sees stakeholders building their own once they trust the data.

2026-03-28 · open the original ↗

• A data story needs four checks, and titles alone must carry it. The presenter's checklist is the so-what (the finding stated in one sentence, a finding not a label), the stakes for the audience rather than for the analyst, the evidence (two or three data points, not a pile of charts) and the ask (a clear action or decision). The test is to read only the slide titles: "revenue overview" says nothing, while "revenue dropped 7% and display ads are the cause" gives the finding, the magnitude and the culprit.
• The audience changes the depth, not the checklist. The presenter's guide is four to six slides or a one-pager for a VP, leading with the yes-or-no decision; six to eight slides with some methodology for product and engineering leads; and the most detail for one's own data team. The figures are averages that move with the time available.
• Bad decks are all situation, with no implication or ask. Every story should run situation, finding, implication (revenue, risk, opportunity) and ask. Cutting is the hard last mile: a chart that took hours goes to the appendix if it does not move the story, and stays only if it supports the finding, answers an objection the audience will raise, or makes the stakes concrete.
• The presenter's Claude Code agent graded a bad deck D and rebuilt it. The system is a "presentation doctor" agent with a deck parser, a deck critique that scores each slide on the four checks and flags anti-patterns, a story extractor that finds one to three buried stories, and a slide transformer: two agents, three skills and one helper file for Markdown, with separate creator and reviewer agents for Google Slides and Docs because Claude Code lays those out poorly on a first pass. On a 14-slide Hawaii tourism deck it gave the title slide 0/12, the deck an average of 2.1/12, 0 of 4 checks met and no ask, and found the buried story: domestic spend dropped 11% and makes up 78% of revenue. Rebuilt for product and engineering leaders, the deck came down to about seven main slides plus an appendix, with finding-led titles and recommendations framed as hypothesis, experiment, cost, impact and confidence. The demo ran on a Markdown deck exported to PDF, not on Google Slides, and was split into steps for teaching, though the presenter says one prompt can run it all.
• The hosts advise building bespoke systems, not fixing generic friction. A host argues that frontier labs will solve obvious pain points such as Google or PowerPoint integration, so time is better spent customizing systems to one's own stakeholders; a co-host mentions a leak that morning of an Anthropic model called "Mythos" said to be a step change over Opus. In Q&A the host says that for individual use a $200-a-month subscription beats the API, which by his own math would have cost about $5,000. He also has Claude write a journal entry to a git-ignored YAML file at milestones before each commit, so context can be cleared and old work resumed. The presenter writes skills only for workflows where Claude needs handholding, and says Sonnet needs more of it than Opus.

2026-03-26 · open the original ↗

• The same model makes better charts inside a harness of chart skills. Shane ran identical prompts on a Kaggle Spotify data set of 953 songs in vanilla Claude Code and in the lab's open-source AI Analyst repo, which carries visualization standards, chart helpers, themes (minimalist, New York Times, Economist, corporate) and a design-review step. The harness version highlights one bar or line and grays the rest, puts the takeaway in the title, labels directly and uses an off-white background. On the first chart the system skipped its own workflow; asked for an ASCII diagram of what it should have done (visualization pattern skills, a chart maker agent, a visual design critic agent that approves or sends back for revision), it named what it skipped and redid the chart. Shane says that outside a demo he would also ask it why it strayed and have it fix the gaps.
• The system picks the story, so it can pick a different one. On audio trends from 2010 to 2023, the harness titled its chart "popular music is getting sadder," with valence down 18 points since 2014, while the vanilla chart showed three colored lines with markers and gridlines. Shane notes the run he did the day before led on danceability instead, that the harness zoomed to a 30–80% axis he would rather start at zero, and that one prompt re-centered the chart on danceability. He uses it for exploratory analysis to find stories, not to send charts straight to stakeholders.
• In the demo, the most-streamed songs were less danceable, not more. The harness reported a Pearson correlation of about −0.11 between danceability and streams; the top 50 songs were less danceable, less energetic, lower in valence and more acoustic than the rest. Danceability of new releases rose 20 points from 2015 to 2023, and 40 of the top 100 songs were outliers below 55% danceability or with low energy. Its three-sentence recommendation to the fictional head of content was to license emotionally weighty mid-tempo tracks with danceability of 45–65% and valence below 50%, against new releases converging at 71%. Shane says the agents are built to go context, tension, outliers, recommendation.
• The analyst stays accountable, and only code creates the numbers. Shane's rule is that the LLM never produces a figure: every number comes from Python or SQL that is saved beside its chart for review, which removes hallucinated numbers but not wrong code. He also backtests the system on past analyses with known answers, which he says has turned up errors in his own earlier work. A live export of the analysis to a Google Doc over MCP first created a new document by mistake, set all body text in bold, and put the code under each chart without the collapsible toggle he wanted; he says Notion handles that toggle well.
• Bundle skills that are used together and split those reused alone. That is Shane's rule of thumb in Q&A, along with asking Claude whether its skills pushed it off course. A co-host describes a skill as a plain-English Markdown file of reusable instructions that works across Anthropic's products, including Claude Cowork and chat. Shane moved the repo's output from decks to documents to make room for the code, and uses HTML that updates as it goes for long-running exploration.

2026-03-21 · open the original ↗

• Treat an experiment as a lifecycle, not a traffic light. The presenter's complaint is the "green means ship, red means kill" dashboard read; the lifecycle runs why, design, power, run, analyze, decide and ramp. A hypothesis takes the form "we believe this change will impact this metric because of this reason", where the "because" forces a mechanism, and pairs one north-star metric with one guardrail. The running example, on a data set the presenter generated, tests regional playlists on a fictional "Streamflow" service, with streams per user as north star and churn rate as guardrail.
• Skipping power analysis leads to peeking and unreadable nulls. Using the usual 5% significance and 80% power, the design came to 6 weeks and 25,000 users, since 12,500 per arm hits exactly 80% power at a 5% effect; a 5% lift needs about 6 weeks and a 3% lift 16. Without it, the presenter says, teams run too short, ship noise after spotting significance on day three, and cannot tell a true null from an underpowered one. The free repo's design skill generated the feasibility curve and power-sensitivity matrix shown.
• Statistical significance is not a ship decision, and segments are skipped most. While running, check sample ratio mismatch and novelty effects; at decision time, weigh practical significance against added complexity. Changes that affect the wider product should ramp gradually (5%, 25%, 50%, 100%) and keep a holdout, because, the presenter says from experience, engagement wins can show up in revenue only over about six months. Of eight analysis questions, the segment cut is the one teams rushing for quarter-end wins most often skip.
• The demo's 14% "winner" hid a churn risk for new users. Run step by step in the Claude web app, the analysis passed the ratio-mismatch, balance and duration checks, flagging weeks five and six, then found a statistically significant 14% lift. The segment step found the churn guardrail degraded for new users while existing users improved, with a Simpson's paradox noted. The ROI step put the existing-user gain at 53,700 users and $11.8 million of lifetime value, still 21 times infrastructure cost when the churn benefit is zeroed out, and recommended shipping per segment. The presenter warns that undocumented column names get guessed at, and that some per-segment calls, such as excluding the Pacific region, may be impractical: the system does 80–90% of the work and a person makes the call. In Claude Code one prompt chained an experiment analyzer agent, an experiment readout agent and a Google Doc creator agent; the full workflow was not yet in the free repo, and the deck export was still rough.
• Validate against past ground truth and fix the system on errors. Shane says the human is accountable for AI results: he backtests on past analyses and puts the Python and SQL into Notion beside the results. With, say, 300 past experiments as ground truth, he suggests Karpathy's "auto research" loop, which repeatedly scores the system, analyzes its errors and rebuilds its skills and agents. The presenter would make area or metric owners accountable for the skills' context, and review junior analysts' output until 95–99% of reviews pass. On a mistake, both say to ask Claude why it erred and what in the system must change.

2026-02-28 · open the original ↗

• Automate the mechanics to flip analysts' time toward thinking. Shane estimates analysis work at about 80% mechanics (queries, joins, validation, chart formatting) and 20% thinking, and argues the aim is to reverse that in the same hours, not to cut staff. He says the lesson was moved up about a month after testing Claude Opus 4.6, which in his account can hold the whole analytical workflow rather than needing step-by-step handholding. It still strays, but its mistakes are now correctable, like a GPS watch flagging a wrong turn, so the bottleneck shifts from execution to judgment.
• The pattern is an onboarding file, agents, auto-firing skills and corrections. CLAUDE.md is the analyst's onboarding doc, reread every session. Agents are prompt templates with rules, skills are guardrails that fire automatically (date-column checks, zero denominators), every number goes through SQL or Python so "the LLM never does arithmetic", and validation is a pipeline step with four layers that halts on failure. Corrections are stored and checked before the next query. Shane says the team's open-source repo of about 40,000 lines compressed into a 1,500-line markdown "seed" that rebuilt the system from scratch on a separate machine.
• The live demo traced a satisfaction gap to one app release. On a synthetic Amazon-like dataset (50k users, 13 tables, a year of data), Claude Code reported NPS of +42.3, free users at +30 against Plus members at +54, and noted that the detractor comments were templated synthetic data. Cross-checking support tickets found a payment-issue spike; parallel subagents ruled out an experiment and pinned it on iOS v2.3, released June 1, with 465 tickets, all of its payment tickets from iOS, running about three weeks. A reviewer agent fixed an overlapping chart label, and the pipeline built a deck with three recommended actions. A separate overnight run on a four-point Q1-to-Q2 conversion drop took about 45 minutes and found a mix shift (Simpson's paradox) rather than a broken funnel.
• Only you can automate your own job, because you validate fastest. Shane argues you catch a wrong date column in seconds on data you know, while anyone automating someone else's domain needs that person's feedback, which for customer-facing tools means a full eval system with ground truth. He likens setup to onboarding a junior analyst, with corrections dropping off by the third or fourth analysis. His advice is to start with a CSV whose answers you already know, and to stay accountable for every number "just like we're using a calculator", logging its queries to a Notion page.
• Checkpoint outputs, bootstrap context from query history, and keep data hosted. In Q&A, to avoid rerunning everything for a small edit, Shane has each agent write results and queries to markdown checkpoints; a colleague switched output to PPTX for manual edits. He bootstraps context on 50–60 tables from 30 days of his own query history and has Claude draft requirements for semantic views. For security he cites zero-data-retention terms with Anthropic, Snowflake Cortex, or a hosted environment such as SageMaker rather than a laptop. He deliberately limits it to product analytics, not modeling or causal inference, which he says needs more human thought to validate.

2026-02-20 · open the original ↗

• Multi-step AI features leak users out of the treatment. In Shane's illustrative case, an internal AI analyst V2 is tested on 1,500 daily users. Of the 743 assigned to it, 156 never open the tool, 190 hit cached V1 results, 129 time out to a V1 fallback and others fall back on errors, leaving 167 (23%) who actually got V2. The headline +3% (p = 0.15) hides roughly +14% among those exposed, but with 167 users against a power calculation needing 760, the test has to be fixed and rerun rather than filtered after the fact.
• AI tests need far larger samples than the usual calculation suggests. Shane says a diluted effect needs disproportionately more users ("half as loud" means listening four times as hard). Non-deterministic outputs add variance, and performance that differs by segment, such as simple versus complex queries, needs a full sample per segment. Because these multiply, he says a test can need around 10 times the users expected.
• Measure correctness with code where possible, and calibrate offline evals. "SQL success rate" can mean valid syntax, executed, non-empty or actually correct. In the example, checking V2 against 200 finance-maintained reference answers finds 5% confidently wrong (off by more than 10% behind a convincing chart). An LLM judge that agrees with humans 80% of the time is a "noisy pen" that washes out real differences, so Shane's rule is to use deterministic code checks for everything else. He argues that over 5–15 experiments, offline quality gains can be calibrated to online outcomes such as task completion.
• More guardrails and pre-registered segments change the ship decision. Shane lists guardrails beyond latency and crash rate: cost per query, safety and PII violations, hallucination, confidently-wrong rate and data leakage, each needing a measurable online mechanism, with quality gains often costing latency or money. Averages hide failures, so the call may be to ship for simple queries and hold for complex ones; segments must be chosen before the test to avoid sample-size shortfalls and p-hacking.
• Run AI tests longer, because users change what they ask. Beyond novelty effects, Shane argues that users who come to trust an AI analyst move to harder questions, so a query mix of 80% simple / 20% hard may shift to 50/50 after launch. He has no fixed rule but suggests four weeks or more and a 5% holdout for a couple of months. In Q&A he describes his team's "shadow mode", running V2 silently alongside V1 on production traffic for realistic quality metrics, which he would build earlier next time; his legal-AI team also leans on causal inference and opt-in beta users because lawyers lose trust fast.

2026-02-19 · open the original ↗

• Measure one true north outcome and the input levers that move it. The presenter, one of the co-hosts, opens with a cautionary case: daily active users rise 30%, the team builds on it, and three months later the gain turns out to be bot traffic. The fix offered is the "true north plus inputs" framework, which the presenter attributes to Airbnb, Slack and Netflix: the true north is the outcome, as close to customer value as can be measured, and inputs are the levers teams pull. Shane adds that you should start from what the person is trying to do with the product imagined away, then move "south" until there is a signal you can measure or build.
• Good true north metrics capture delivered value, not attendance. Daily active users, revenue and signups are given as bad choices, and Airbnb's nights booked (traveler housed, host paid), Slack's messages sent and Netflix's hours watched as good ones; Netflix is said to have moved there from subscribers added. A seven-test checklist splits strategic tests (reflects customer value, leads to business outcomes, hard to game, one rallying number) from practical ones (actionable, understandable, measurable). Ratios such as delivery success rate, and cumulative totals that can only rise, are ruled out as true norths.
• Decompose the true north into breadth, depth, efficiency and frequency. Retained signups breaks down into traffic × signup conversion × retention, each owned by a different team. Shane adds that the true north exposes gameable inputs such as bot or one-off paid traffic. In the B2B exercise, a project-tool PM's monthly active users becomes team projects completed per week, with inputs of active teams with a shared project, tasks completed per project, time from project creation to first task, and weekly active collaborators. In the food-delivery poll the answer is orders delivered per month.
• Expect to revise metrics, and locate inflection points in the data. In Q&A the hosts say a new metric tends to change every couple of weeks for the first one to three months; the time window should follow the product's natural cadence, from daily to six-monthly; and there is no fixed cap on how many metrics to track, so prioritize by leverage. Shane describes regressing an input (AI output quality) against a true north (time saved) to find the point where gains accelerate or flatten, which he treats as the "aha" milestone worth setting a goal on.
• The hosts say a new model made their AI analyst workable. They credit Claude Opus 4.6. The metric-coaching prompt demo was skipped for time; instead the hosts showed decks from their Claude Code AI analyst — on a co-host's real estate business, Hawaii tourism data, and an interactive HTML campsite deck with presenter notes — which they say involved no hand-written code or slides. Shane describes a repo of about 60,000 lines and about 20 agents, and an overnight test in which a single markdown "seed" file had Claude Code rebuild a data team for a new use case. His hedged view is that analysts will move toward building systems of agents and toward technical-PM roles.

2026-02-19 · open the original ↗

• Analytics leaks value between the dashboard and the decision. Nate Nichols, vice president of product at Tableau, describes analytics as a funnel where only the last stage, a better or faster decision, produces money, and calls the dashboard-to-decision step the lowest-conversion stage, with roughly 30% of an organization looking at data regularly as a long-standing ceiling. He argues the data team's job is driving outcomes rather than delivering requested dashboards. His own position, not necessarily Tableau's, is that success should be measured on outcomes, though he says nobody has found that metric.
• Pull semantics out of dashboards so humans and agents agree. Nichols says Tableau has moved definitions into a metrics or semantic layer, so each user can slice a metric their own way and the system can monitor on their behalf. His forward-looking third point is that agents working around the clock must see the same numbers, governance and data models as humans, or "that's like chaos".
• Trust is earned by clarifying, showing reasoning and testing. Nichols says LLMs are "dying to be useful" and will silently pick a meaning for "best salesperson", so Tableau asks first, shows the reasoning and data model, and offers analysts a testing center that proposes 20 questions per data model for approval, plus an AI-readiness score. He cites Tableau Pulse, whose Q&A answers only from deterministic, pre-computed insights rather than generated SQL, as among the first gen-AI features many customers got through security and legal review.
• The demo ran from question to action in one conversation. On Tableau Next, which runs on Salesforce's Agentforce, the agent asked what "lowest performing" meant and said it excluded smart home per the organization's preferences. A beta proactive alert explained a 4% drop by a sharper fall in California. A Slack canvas showed live metrics and every conversation about win rate, and an agent was told to have reps book executive business reviews. Nichols says tracking whether such actions work, his example being a 14% lift in close rate, is "not quite there yet".
• Analyst roles will split between plumbing and the business. Nichols expects some analysts to move toward data engineering and enriching the semantic layer, and others to embed with business teams and coach the agent on team-specific context. He cites a Tableau Next customer that went from 10% to 80% weekly analytics engagement, and frames AI as an "Iron Man suit" with the analyst's judgment still in charge.

2026-02-12 · open the original ↗

• A good analytical question passes three checks. The presenter, one of the co-hosts, sets out a question quality checklist: decision-tied (what would you do with the answer?), data-grounded (can data actually answer it?) and specific (bounded, with a clear point where it is answered). The anti-pattern is "it would be interesting to know", and the lesson's premise is that, for humans or AI, question quality sets output quality.
• Sharpening a question means naming the decision and a threshold. The worked progressions run "why are users churning?" to "which first-30-day behaviors correlate with churn so we can trigger interventions"; "how are users behaving?" to "which three features correlate with 30-day retention above 50% for last quarter's users, and should we surface them in onboarding?"; and "how's marketing performing?" to "which channels exceeded our 1.5x return-on-ad-spend target last quarter, and should we shift budget?". An attendee points out, and the presenter accepts, that these assume the company agrees on what churn means.
• Even exploration needs a direction. To the pushback "I'm just exploring", the presenter offers reframes: "what patterns would change our next sprint priorities?", "what would I need to see to recommend we invest in X?", and "what's worth investigating further is your decision". Undirected exploration is "a fishing expedition"; directed exploration is "discovery with purpose".
• Use AI as a question coach, not a grader. In a ChatGPT demo the presenter pasted a coaching prompt and a vague request about a spike in support tickets; the AI walked through each check in turn, and the answers supplied (high-priority bug fixes, ticket volume by day, a 10% week-over-week spike threshold) produced sharper questions such as which ticket categories drove the increase over the past two weeks. The argument is that a conversation shows why a question is weak, building the skill before delegating to AI.
• Ask the requester what decision their question drives. In Q&A the presenter says that, in their experience, about 60% of the time the requester then concludes it isn't a good question and drops it, which they call high-ROI. Asked by an attendee whether to frame questions differently for humans than for AI, the presenter says the structure is the same but people need to be met where they are; a third co-founder adds that you should explain why you are questioning the question so it doesn't read as resistance.

2026-02-11 · open the original ↗

• Ticket-based BI turns data teams into "human APIs". Ramy, co-founder and CEO of Querio, recalls that at Amazon a ticket to the central BI team averaged six weeks, while Ramy, a technical product lead who knew the data flows, could answer in an hour. At that scale, Ramy says, the data team no longer understood how the business mapped to the data, and template queries carried miscounts. Ramy and a host agree the fix is to empower the person closest to the business or to embed the data person deeply.
• The bet is transparent BI-as-code fed by a curated handbook. Ramy describes four components — a notebook, agents, context/governance and a front end — with SQL in, table out and Python visuals, "no black box". Ramy likens context to the best analyst on earth on their first day, rereading one handbook of joins, catalog, metrics and quirks for every question. In the Dunder Mifflin demo, a vague "compare 2024 versus 2025 sales" prompt noticed data only ran through September, compared January 1 to September 15, fixed its own syntax error and ended with key insights.
• AI automates the easiest part of data work, not the job. Ramy argues board and exec metrics still need manual care and executive trust, and that definitions, context upkeep and data cleaning remain the hard part. The example is a telecom customer whose gigabytes column stores unlimited plans as negative one, which "no AI model on earth is going to know". Ramy also claims zero-shot text-to-SQL on a decent schema was only solved about three months before recording.
• Prove value in a month on half the tickets. Querio's pilots, Ramy says, use about 10 eval questions spanning tables that need specific knowledge, an agent first pass at the catalog and weekly context updates. The goal is that marginal effort solves half of incoming tickets, tried by the data person before the team gets it. Because most data roles go unfilled, Ramy predicts unfilled roles simply won't need filling rather than people losing jobs.
• Self-serve users ask more and learn how data works. Ramy reports that users don't actually care about dashboards, ask far more questions, and come to understand table choice and many-to-one relationships, while data people shift to reviewing drafts users send them. Ramy argues that investing in the semantic layer now pays back directly in less busywork, and that analysts should learn how the business flows into the production database.

2026-02-09 · open the original ↗

• Agents and humans must work from the same numbers. The speaker argues that human–agent collaboration breaks down if an agent queries Snowflake while a person reads an extract and the figures disagree; an agent optimizing marketing spend around the clock, or recommending products to a calling customer, should see the same governed data, security and data models — the same product list a human sees in a dashboard. They predict this becomes "really important over the next few years" as more agents work alongside people.

2026-02-09 · open the original ↗

• AI advances faster than expected, even when you expect that. The speaker offers this as a self-styled "law of AI", pointing to ChatGPT's 3-year anniversary: reasoning, agents, image, video and audio generation and understanding, and runs of eight hours on advanced math problems were not in view three years ago, when people were making limericks and mocking images with extra fingers. They say predicting three years ahead is hard, beyond expecting it to be impressive.

2026-02-09 · open the original ↗

• Data is the last transactional relationship at a company. The speaker likens it to asking someone to type for you: the person doing the grunt work is not the one who needs the insight, so data teams become "human APIs". Their alternative is either to empower the person closest to the business to get data themselves, or to embed the data person so deeply in one area that they hold the context and the back-and-forth disappears.

2026-02-09 · open the original ↗

• Agents are the abstraction that finally lets end users do analysis. The speaker describes Querio as BI-as-code built from the ground up for agentic analytics: each step from machine code through assembly and C++ to Python abstracted further, and agents let a user prompt in plain English while the agent writes the code. Their claim is that this wraps away the database, schema and code that BI tools never solved for end users.

2026-02-04 · open the original ↗

• Users want plain dashboards, not the data world's vocabulary. Javier, a founder of Index, says that after talking to close to 500 people, there is a gap between conference and investor talk (semantic layers, metric definitions) and what people use: bar charts, sheets, dashboards. Customers keep saying they do not want notebooks and have no mental model for them, so form factor sells. Javier adds that polish matters but, as another founder put it, is "necessary, but not sufficient".
• The next version is chat on the warehouse, with proactive questions. In Javier's account, Index 1.0 is a dashboard plus a SQL data studio with plain-English query writing. The previewed 2.0 suggests questions by scanning a connected source before the user asks, and shows the query, lets the user edit it and previews the underlying table. A demo thread took under two minutes. Javier positions it to sit beside Looker or Metabase rather than replace them, because asking a firm to migrate 300 dashboards is "a huge ask".
• Bad data defeats the model, so human judgment stays. On a cleaned Kaggle NBA dataset the AI reported 219 championships because it confused playoff games with titles, even with reasoning on. It also mishandles dates (UTC offsets, year-to-date returning December) and flips bar charts. Asked what a company needs first, Javier invokes the Pareto principle: data quality, with everything else "a variation" of it. Javier likens the product to the year or two before Waymo was fully self-driving.
• SQL will matter more than ever, but be more abstracted. Javier argues SQL is the lingua franca everything resolves to — Index 2.0 connections to Stripe, Salesforce and HubSpot are all queried as SQL — while users stop caring about it. The durable skills are judgment, understanding goals and metrics, and knowing which questions to ask, without which Javier warns of "analytics slop".
• The next step is acting on data, not just reading it. Beyond proactive insight, which a host also predicts, Javier forecasts tighter interconnection: emailing your top 10 leads straight from the data tool, which Javier says is neither reverse ETL nor the current paradigm. The argument is that this collapses the days-to-months lag between an insight in a deck and someone acting on it, and turns data people into more embedded, action-oriented business roles.

2026-02-02 · open the original ↗

• Data quality is the precondition for agentic analytics. The speaker argues that teams told to "use more AI" will find it is not a switch: with bad data agentic analytics will not work, just as ordinary analytics will not. Invoking the Pareto principle, they suggest nearly every other requirement they could name is a variation of data quality.

2026-02-02 · open the original ↗

• AI analytics should move from reactive to proactive, like an analyst. One speaker describes a future where the system surfaces what you need, knows its own limits and asks humans what to do next, leaving people to judge whether it makes sense. Another likens this to an analyst's career: reactive and told what to do, then connected with the team, then proactive — anticipating questions the team would never have thought of.

2026-01-31 · open the original ↗

• Data scientist pay at Facebook climbs steeply with each level. The speaker gives average annual pay of $157,000 at IC3, $250,000 at IC4, $370,000 at IC5 and $515K at IC6; the IC7 figure is cut off mid-clip ("around $750").

2026-01-31 · open the original ↗

• The system around an LLM, not the model, creates the value. The speaker says LLMs were once treated as magical but cannot do everything themselves; the model acts as the brain or focal point for decisions, and the surrounding pieces, which did not get the credit they deserved, are what make it valuable.

2026-01-31 · open the original ↗

• Struggling to understand abstractions made the speaker a good explainer. The speaker says they were always slow and found high-level abstraction hard, through grad school and work as a mathematician, programmer and data scientist, so they had to tell themselves stories to understand things; after several career changes they decided explaining was what they did best.

2026-01-31 · open the original ↗

• Senior data scientists can out-earn many doctors. The speaker puts family medicine and pediatrics at around $250,000 a year, the same as an IC4 data scientist, says staff-level data scientists earn more, and sets high-earning specialists (neurologists, anesthesiologists, surgeons) at around $800,000 against $750,000 for a principal staff (IC7) data scientist at Facebook.

2026-01-31 · open the original ↗

• Jailbreaking and prompt injection differ by who is in the exchange. The speaker names prompt injection, jailbreaking and indirect prompt injection as the top three attacks: jailbreaking is just a user tricking a model into saying something bad, while prompt injection adds a developer instruction (such as "translate the following user input") that a malicious user's input tries to make the model ignore. The clip ends before indirect prompt injection is explained.

2026-01-31 · open the original ↗

• Choosing data engineering starts with knowing your own temperament. The speaker says someone who communicates and partners well and can sit through long sessions of cleaning data may be a good candidate for data engineering.

2026-01-31 · open the original ↗

• Analysts who find the right problem won't be replaced by AI. The speaker argues that data scientists and analysts who focus on what matters most to the business, apply critical thinking and ask the right questions can use AI to get more done and deliver much larger impact.

2026-01-31 · open the original ↗

• Critical reasoning will be the most underrated skill alongside AI. The speaker picks the ability to reason, contextualize and synthesize information with AI, using intuition and creativity rather than relying on AI fully, because human judgment is still needed.

2026-01-31 · open the original ↗

• The clip poses its question and cuts off before the answer. A host asks the others whether their reason for doing the podcast changed over its first year; the co-host who answers gets only as far as "the answer for me is an unambiguous" before the clip ends.

2026-01-31 · open the original ↗

• Keep making breaks rather than hunting the optimal route. The speaker says everyone looks for the most efficient path to a destination, but making breaks happen creates opportunity, and you are then ready to seize it when it appears; the clip ends mid-sentence.

2026-01-31 · open the original ↗

• Executives care about business impact, not your F1 score. Shane sets three levels: shipping blind with no quality metric (feedback arrives from angry or churning users), quality-confident without knowing the effect on revenue, and able to tie AI changes to business outcomes. He says most eval content stops at level two, and even large companies reach level three for only some features.
• Every link in the "AI impact chain" must hold. The four links are feature change, AI quality, customer value and business impact. Quality metrics (correction rate, acceptance rate, error rate) are not customer value (time to finish a task, task success rate), and customer-value metrics like engagement can mislead: his own browsing for a pizza peel looked like engagement but was confusion. Revenue alone invites false credit; at Nextdoor, he says, engagement spikes teams claimed were repeatedly Texas ice storms.
• Customer value breaks ties that quality metrics cannot. In his illustrative text-to-SQL example, V2 adds planning, schema context and a validating agent: latency rises from about 30 seconds to two minutes, edits per query fall from 2.4 to 0.3, and quality can split by segment (he posits better on customer-support questions, worse on finance, leaving those teams disagreeing on whether to ship). Time from question to chart export drops from 12 to 3 minutes, which settles it; he adds that enough A/B tests let you learn which quality metrics actually drive customer value.
• Causal inference converts customer value into forecast revenue. In his made-up worked example, a model coefficient of five seats per minute saved, times 9 minutes saved, gives 45 seats per account; across 100 accounts at $200 a seat that is $900,000 in MRR, about $10.8 million ARR, with an interval of roughly $8.6–13 million. He uses AI quality to decide what to test, customer value to decide what to roll out, and business impact to decide what to invest in.
• Without an A/B test, backfill quality metrics and model causally. In Q&A he suggests backfilling about six months of quality metrics for causal analysis when users are too few or the industry too risk-averse, focusing on quality plus qualitative research before you have users, and agreeing latency and cost guardrails up front. He recommends vibe-coded annotation UIs that mirror the product (redline diffs, a live chat pane), and says his own eval pipeline took three or four months to build.

2026-01-31 · open the original ↗

• Product metrics exist to support decisions, not as goals. Shane says a metric should be tied to a real decision (ship, roll back, invest, stop), so a wrong metric costs exactly a wrong decision. Users come to a product for a real-world job, such as booking a trip to Hawaii, not for dwell time, and two users with identical in-product behavior can end with very different outcomes.
• North star metrics bridge gameable inputs and lagging business outcomes. Input metrics are frequent and measurable down to the millisecond but easy to game (spamming more emails raises email clicks without adding user value), while revenue or churn can take months to move. The north star ties day-to-day input metrics to user value and business outcomes; he defers how to prove that causality (correlation, A/B tests, causal inference) to a later lesson.
• AI breaks the deterministic-output assumption behind traditional metrics. One input can now produce many outputs, so a demo that works on a few inputs fails unpredictably at scale and teams hear about it from customers, often too late. His view is that the unit of observation shifts from the model's output to the output as experienced by the user, and what to evaluate (correctness, faithfulness, completeness, safety, tone) depends on the product.
• Start from the user's workflow, then turn it into hypotheses. His process: name the real-world job and what good and bad look like, then state "if AI quality increases, we expect X, resulting in Y user behavior". He prefers ground truth as close to the user as possible (historic user data, in-product signals), falling back to subject-matter experts, authoritative sources or rules, and says annotation is easiest in a UI that mirrors the product, such as a chat interface rather than chat flattened into a spreadsheet.
• Use several metric types, then prune to a few key quality indicators. Structural checks, similarity scores (ROUGE, BLEU or bespoke) and judgment checks (human rubrics, model judges) each miss things, so you will want more than one. He filters them by a logical, if not proven, link to the north star, and treats the whole loop as continuous because new inputs keep producing new failure cases.

2026-01-28 · open the original ↗

• Data work fails on fragmented context, not on writing SQL. The guest, LiveDocs' founder and CEO, says the business context and the technical skill rarely sit in one person, so product managers and data scientists ping-pong over requests. An LLM has to be given the same context a human data scientist would need, which in their view exposes the real problem rather than solving it.
• The founder's "AI data scientist" moves from assistant to proactive researcher. Stage one, as they describe it, gets serious data work from zero to 80% in seconds by removing the "yak shaving" of environments, credentials and cleaning. Stage two, which they say LiveDocs is building now, has the agent act unprompted (having measured churn, it proposes and runs a retention analysis), pull models from Hugging Face and send pull requests to a dbt repo, with a human in the loop. They predict some data-ingestion incumbents will not exist two years from now, because LLMs can write one-off ingestion scripts.
• The demo shows a reactive notebook the agent works inside. The founder says LiveDocs cells mix SQL, Python, charts, inputs and LLM prompts and export to plain Python or ipynb. In the demo the agent committed to git, explored a CRM export CSV and named the top-performing branch; a slider input drove a reactive query and chart published in app mode; a notebook was scheduled for 9 a.m. with email, Slack or webhook delivery, and notebooks can trigger one another.
• Wrong measurement is the hard error; failed queries are easy. The founder says failed queries are caught via dry runs (BigQuery, for example) and retried, but a query that measures the wrong thing needs a human with context. Their answers are uploading docs and data glossaries, an explain-the-query feature in a sprint that week, a pull-request-style handoff for data work, and an indexing "coverage" percentage that they say correlates roughly with agent quality.
• Prompting is the top skill, and schemas are the wrong context. Counterintuitively, the founder says pasting schemas and data types wastes the context window because tools infer them at runtime; describing relationships and references between tables helps more. They also advise reviewing output against the objective, picking models by task (stronger coding models for analysis, faster ones for an email), and taking AI output with a pinch of salt, at least for the next year.

2026-01-21 · open the original ↗

• BI tools solve a two-decade-old problem: getting data into the business. Olly Hughes, co-founder and CEO of Count, says organizations now have too much information rather than too little. His hypothesis is that the BI tool is among the most expensive and lowest-ROI software a company buys, and that dashboards bake in a support-function role for the data team.
• Most data-team work only maintains the status quo. Hughes says he interviewed 400 data leaders, who told him about 90% of their teams' work produces no incremental value, though the best teams reach 50/50 or 40/60. He names four "service traps": drowning the business in information, acting as an answering machine (which AI could turn into a "Cambrian chart explosion"), retreating from stakeholders, and building things the business cannot see, such as dbt tweaks or Snowflake cost tuning. His simplest test is whether the exec team can name three examples of the data team directly driving value last quarter.
• The data team's purpose is to visualize the business, not data. Hughes argues the way out is "operational clarity" (making the business look simple, with a high signal-to-noise ratio) and being the organization's problem solvers, working in agile iterations rather than waterfall. A co-host adds that the quality of stakeholders' questions decays when they only ever get charts back; Hughes says purpose is what lets a team say no to low-value questions, and names time to decision as a metric to track.
• The proposed answer is one shared canvas for analysts and stakeholders. Hughes describes Count as an infinite collaborative whiteboard, like Miro or Figma, running on live data. The demo showed a metric tree (revenue broken into MRR, churn and upsell), a metric map of about 60 onboarding-funnel metrics, and SQL cells whose downstream metrics are traced and that can be pushed to dbt or a semantic layer. He says customers also use it as a dbt IDE.
• His agent works on the canvas, where every output stays editable. In the demo, an ambiguous question about Chicago led the agent to break the problem down and report that Chicago ranks seventh in sales but loses money through excessive discounting, mainly in office supplies, while a narrow query ran in parallel. Hughes says AI amplifies whatever paradigm it is given and that teams can work this way without Count; Shane adds his view that nondeterministic AI products make data science more important, not less.

2025-12-22 · open the original ↗

• AI output quality can't be checked with rule-based tests. The speaker argues that traditional software testing works because a deterministic function can check whether a goal was met. Accuracy, hallucination or the quality of a legal draft is subjective and has many dimensions, and none of them reduces to a simple automated check.

2025-12-22 · open the original ↗

• AI-powered analytics and AI product evaluation are the next focus. Shane names these as the lab's two themes for 2026 and says the themes recur with every guest. One is AI letting people make data decisions with less reliance on a data person. The other is that evaluating AI products needs different tools, a continuous loop, and input from researchers and subject-matter experts, which he thinks data scientists, analysts and ML engineers are well placed for. He announces a 2026 interview series with the founders and leaders building each kind of product (as of December 2025).

2025-12-22 · open the original ↗

• The clip asks a question and stops before it is answered. A co-host asks whether the hosts' reason for running the podcast changed over its first year. The first respondent says "an unambiguous yes", and the clip ends before giving a reason.

2025-12-18 · open the original ↗

• Recording is the easy part; editing and distribution eat the time. The three Data Neighbor co-hosts, who met at Nextdoor, say that even a rough edit or a bad thumbnail takes longer than it should. Shane, who does the thumbnails, says no one warned him about the YouTube work. A co-host says the weekly back-office work of publishing to Spotify, Apple, Substack and LinkedIn takes far longer once it compounds.
• Busy, well-known guests say yes if you are consistent and courteous. A co-host says a prominent statistics educator agreed to appear when the show had nothing to show, and that they spoke to 10 CEOs and co-founders in the month before recording, some from large companies. A co-host says the free advice guests give would otherwise cost thousands.
• The podcast's purpose shifted from helping pivoters to learning AI. The co-hosts started out to help people move into data, but a few episodes in they decided the AI shift was faster and more consequential than was being discussed. They say they ran out of their own knowledge in about three episodes and now use the guests to upskill themselves. Shane's turning point was an episode on building products with Replit, whose guest teaches paid courses on it.
• Three co-hosts was a deliberate hedge against burnout. A co-host says the odds of one person burning out or quitting are high, so two is better than one and three lets them rotate. Shane adds that each episode has an informal lead and a support, so the second host can always fill a lull in the conversation.
• At principal level the job reaches beyond your company. A co-host who is at principal level describes a ladder: a junior is handed work, a senior IC tells the manager what the team needs, staff works across teams, and principal must bring the industry's practice in and push the company's out. A weekly podcast, the co-host says, forces that outside contact. The co-hosts also credit it with more visibility, more inbound recruiter messages and more structured speaking, which one co-host estimates has improved 10x to 20x.

2025-12-10 · open the original ↗

• Enterprise research is set by a north star and product needs. Shelby Heinecke, who leads a research team at Salesforce, says it is not academia. Research across the org aligns to a north star the company calls "enterprise general intelligence": getting agents to perform real enterprise tasks, which she says they cannot yet do despite impressive demos. Road maps are also built jointly with product groups such as Data Cloud, and her team's role (model building only, or deployment too) varies by project.
• Crisp scope and an agreed eval set carry AI past demo. Her example is a set of small models shipped to Data Cloud customers a couple of months before recording to unify ingested customer records. They replaced a limited rules-based approach and decide, for example, whether two records are the same person. The scope was narrow, the evaluation set was built with PMs before any work began, and small models were chosen for speed rather than capability. She says training the model only gets you a demo, and ethics, security, legal, PMs and product engineers must all be brought in to ship.
• Research roles need proof you can create new things, not a PhD. Heinecke says a PhD is one common way to show the skill. A traction-winning open-source repo or a substantial side project also counts, as does undergraduate work with a professor's group. A co-host adds that two-week side projects teach a skill but three- to six-month ones mimic the job. She also urges early-career people to network at academic conferences (NeurIPS, ICML, ICLR) and industry ones, and says it gets easier with practice.
• Technology is the means; the customer goal is the constant. She says her team stays adaptable by trying each new model against projects, but there is no fixed solution beyond keeping the customer as the north star. She has real cases where older models work better. A co-host says they now repeat the data team's mission every other week so people do not chase the newest model.
• AI teams will turn interdisciplinary, and agents will become embodied. Heinecke predicts domain experts will join ML teams, giving feedback for reinforcement learning, designing training data or annotating. A co-host notes their legal tech company keeps lawyers in-house for labelling. She expects agent benchmarks (Salesforce has open-sourced a CRM benchmark) to drive agent improvement over the next 6 to 12 months, and agents in robots and drones to reach market in a year or two.

2025-12-10 · open the original ↗

• AI teams will become interdisciplinary rather than model builders alone. The speaker predicts that deploying models to customers and discovering use cases will bring together model builders, people working on discovery, and domain experts who help fine-tune or train models for their domain.

2025-12-10 · open the original ↗

• AI projects reach production with crisp scope and an agreed evaluation set. The speaker describes a project scoped tightly to a known data situation and the fields customers ingest. An evaluation set reflecting real customer data was defined with PMs before any work began and was agreed by everyone, including the product owners. The speaker calls these two the main ingredients of an AI project that makes it to production.

2025-12-03 · open the original ↗

• Community growth comes from consistency, not a formula. Sadie, founder of Women in Data, says you must show up about ten times before people trust you. She ran an event every month for 3 years, demoing something she had just learned when no speaker came. The group began with two people at its first meeting and is now 70,000 people across more than 120 countries, 10 years on. A co-host applies the same rule to a fortnightly insights forum at work: miss one and people start double-booking it.
• Listening to members set up Women in Data's inflection point. Sadie describes about five slow foundational years, reaching roughly 5 to 10 chapters and 10,000 people. In 2019 members asked for something between meetings, so an online platform was planned for 2020 and was in development that January. When everything moved online in March, it enabled global expansion with no change management needed. The mission, she says, moved from her own need for community to increasing diversity in data careers.
• Women's share of data roles falls sharply with seniority. Sadie's figures are roughly 30% among data analysts, about 12% in analytics and ML engineering roles, and 4 to 5% among chief data officers. She argues that focus belongs on leadership because it trickles down. A co-host offers the hypothesis, untested, that teams led by women attract more women, citing their time at Nextdoor.
• AI rewards deep domain expertise; you become the conductor. Sadie says computing is shifting from deterministic to probabilistic and likens the change to going from orchestra musician to conductor. A conductor must understand the music better, so she says people who think AI removes the need for skills have it backwards: the future favours domain depth and the ability to orchestrate people and agents. A co-host says 20 to 30 AI guests have consistently named communication and business context as the durable skills.
• Curiosity is a trainable muscle and the hedge against displacement. Sadie says even if AI could do her job, her open questions would keep her working. To build curiosity, she suggests breaking habits, for example a "color walk" hunting for one colour, because people suffer their own "model collapse". Her new Human Machine Collaboration Institute pursues questions such as what consciousness is, builds AI and robotics ecosystems for smaller cities, and has a partnership with Nvidia and the city of Rancho Cordova that she traces to a former Coursera student.

2025-12-01 · open the original ↗

• List the parts of your job you enjoy, then search for them. The speaker found that what they liked about lab work was collecting data, analysing it and designing experiments. A Google search for those turned up "data science", a term they had never heard. They quit the lab job, took early Coursera courses and got a job working with data.

2025-12-01 · open the original ↗

• Computing is shifting from deterministic to probabilistic, demanding conductors, not players. The speaker says AI long ran behind the scenes in operations and is now in front of the general public. Their analogy is moving from playing in the orchestra to conducting it. The clip ends before the analogy is developed.

2025-12-01 · open the original ↗

• You must show up about ten times before people trust you. The speaker ran a Women in Data event every month for 3 years, doing a demo of something recently learned when no speaker was available. People come to rely on an event that is always there even if they miss a couple of months. The consistency built the community's trust and gave the organiser practice.

2025-11-19 · open the original ↗

• Self-serve BI failed on adoption, not on the tools. Lei Tang, CTO and co-founder of Fabi, describes running a data science team (captioned "lift") with hundreds of tickets a week in the backlog. Mode dashboards he shared went untouched for three months ("a dashboard graveyard"), and weekly SQL training for growth marketers, with office hours and homework, left very few able to write their own queries. He says semantic layers and human-curated documentation cost too much upfront and cannot keep pace with a fast-changing organization: "any tool that requires human curation" fails, except Wikipedia.
• Vibe analytics means the human forms hypotheses and AI implements. Lei's example: when activation drops 10% in a week, the analyst proposes explanations, such as product lines or user segments, and AI writes the window functions and plots that confirm or reject them. A domain expert who looks at the business daily can tell when an output is wrong and can iterate or ask the data team to check it.
• AI should learn the semantic layer rather than wait for humans to write it. Lei says Fabi crawls schema metadata (keys, uniqueness, nulls) and learns from two places: the organization's most heavily QA'd KPI dashboards and users' own interactions. It shows what it has learned as readable markdown for a quick human yes or no, instead of YAML or JSON. A co-host adds that missing logging and tracking, plus undocumented business context, still break AI analysis.
• Assume data quality is bad and begin with guardrails. Lei says no head of data among hundreds of customers and prospects has ever called their data good, and 3–6 month single-source-of-truth projects tend to stall. He recommends seeding a few dozen core tables, which he bets cover about 70% of requests, and giving all tables to technical users while non-technical users see only datasets a data person has published. His warning: a Fabi agent given past conversation histories and tool results made up a successful "dry run" and its preview.
• Critical thinking, not coding, is what protects analysts' jobs. Lei says AI writes SQL that runs cleanly but has the wrong logic, because business logic is undocumented and spread across people. He argues analysts won't be replaced if they focus on what matters most to the business. Deep ML and GPU work will still need strong coding, AI acts as a multiplier that makes analysts "full stack", and Fabi lets candidates use AI in coding interviews.