AI Analyst LabHow to evaluate an AI data analyst's answers in Claude Code
synopsis
This AI Analyst Lab lightning lesson, led by Shane with two co-hosts on the chat, argues that with AI now producing whole analyses in minutes, the work has moved from getting the answer to knowing how far to trust it. It maps the checks an agentic analytics pipeline needs, then runs one of them live, a reliability check. The demo ends on the lesson's main point: an answer that holds still across runs is necessary but not sufficient, and an answer that moves is a metric-definition problem the analyst has to settle with people.
Why it matters. Shane says agentic analytics, a natural-language question answered with SQL, a chart and a recommendation and no human in between, has gone from party trick to real decisions within the past four to six months. When it is wrong it does not look wrong. The costs are lost trust in the tool and the analyst once someone catches it, or, worse, a decision acted on and found wrong a quarter later.
Evals as a dial. He defines an eval as a check on how much to trust an answer you did not work out yourself. It sits on a scale matched to the stakes, not a yes/no stamp. The system is a pipeline of at least seven steps. Errors early in it, such as a misread question or a wrong metric definition, flow cleanly into everything downstream, so those steps deserve most attention. Each step has its own checks, and two run across the whole pipeline: getting the same answer across models, and tracing every number to its query.
Four ways to check. Almost any check is one of four kinds: a known answer, a rubric graded first by humans and then by an LLM judge, asking several ways, or showing receipts. Shane's caveats are that agreement is reassurance, not proof, and that disagreement is the more useful signal. He also says lean teams can stop at the cheaper checks.
Reliability demo. In Claude Code, a not-yet-published reliability skill sent each question to five independent subagents querying Snowflake, then reported mean, variance and stability. Checkout conversion came back 33.2% all five times. Shane says that was only because a metric dictionary added that morning pinned one of several defensible definitions. Retention, absent from the dictionary, came back 99.3%, 99.2%, 9.7%, 34.9% and 78%, each a different real definition. The check caught the instability but could not pick the right reading. That takes agreeing a definition with the team and storing it as context. A hook recording every Snowflake query, with timestamp and purpose, gives receipts that do not depend on the model's memory.
Q&A. For messy data or small teams, Shane says start with reliability and receipts, and look for ground truth already lying around in finance-reconciled numbers and much-run queries. Models sharing training data share blind spots, so agreement between them is not proof. For banks and fintechs that restrict AI, he suggests metric definitions are not secrets and cites anonymization and hosted zero-retention setups; a co-host adds local open-source models. Shane says that inside a harness Opus 4.6 performed on par with Fable. A co-host says the slow part of a semantic layer is the people work of agreeing definitions, not building it. Shane closes by predicting that better models will need more checking, because they will also get better at being convincingly wrong.
- Trust in an AI analysis is a dial, not a switch. Shane defines an eval as a check on how far to lean on an answer you did not work out yourself, with rigor matched to the stakes: a quick look for a curious question, real checking for a number going to the board or the street. Because analysis is a pipeline of at least seven steps (interpret, pick tables, pick metric definition, write SQL, run, read, recommend), he says early errors such as a wrong active-user definition carry cleanly into every chart after them, so that is where to spend effort.
- Four checks cover almost any output. They are a known answer (ground truth), a rubric (refined by humans first, then scaled with an LLM as judge), asking several ways, and showing the receipts — the exact SQL behind each number, which he calls the cheapest and says analysts should do regardless of AI. His caveat: agreement across runs or models is reassurance, not proof, since a system wrong for the same reason agrees with itself, and he finds disagreement the more useful signal because it says where to dig.
- A stable answer can hide an unchosen definition. In the demo, a reliability skill in Claude Code ran "how are we doing on checkout conversion?" through five independent subagents against Snowflake and got 33.2% every time. Shane says the stability came from a metric dictionary pinning one of several defensible readings (checkout start to payment, all sessions, cart to checkout, people versus sessions), so the check showed consistency, not that the definition was the one stakeholders meant.
- An undefined metric produced five different retention numbers. Asking "what's our retention rate?" with no dictionary entry returned 99.3%, 99.2%, 9.7%, 34.9% and 78%, each a real definition (month-after-signup activity, 30-day repeat purchase, two-plus orders over a year, a second session within seven days). The check flags the problem but cannot pick the answer; the fix he gives is agreeing a definition with the team and storing it as context, such as a YAML metrics file, and a hook logging every Snowflake query supplies receipts that the model cannot hallucinate after its context is cleared.
- Better models will need more checking, not less. Shane predicts models will get better at being right and at being "super convincingly wrong". In Q&A he says that inside a built harness Opus 4.6 matched or beat Opus 4.7 and 4.8 and was on par with Fable, which won only with no system around it — his basis for expecting open-source models to catch up; for security-restricted firms he cites anonymized data and hosted, zero-retention setups, and a co-host adds locally hosted open-source models.