Searching thousands of AI-generated models builds optimism into the winner. Borrowing Christian Kästner's "signal collapse" argument (cheap signals stop separating good work from bad), the author calculates that the best of 1,000 candidates has an expected optimism of about three standard errors of a single AUC estimate; with a typical standard error of about 0.01, that is roughly 0.02, the whole gap between a 0.82 and a 0.84 model. Agents that keep adjusting proposals against the same validation sample turn it into part of development ("automated overfitting"), and the author argues selection criteria must be fixed in advance by someone other than the searcher.
More AI-generated validation tests are not more effective challenge. In the author's credit-model example, an AI validator reproduces random-split AUC results and adds metrics, all correct, yet misses thin-file borrowers, calibration that breaks in a downturn, or an edge that vanishes out of time. Development and validation agents sharing data, assumptions and evaluation habits share blind spots, so the author holds that independence has to extend past reporting lines to test design, protected evaluation data, acceptance criteria and decision authority.
Development and validation should run as two separate, connected discovery loops. The author's proposal: development jointly minimizes predictive risk, complexity, instability and a fairness or disparity measure along a Pareto frontier, with the LLM proposing and protected confirmation data deciding; validation searches the other way for segments, conditions or shifts where loss exceeds overall loss by a material margin. Because adaptive weakness search also finds noise, candidate findings need independent confirmation with uncertainty estimates and a multiple-testing correction. A model revised against the same evaluation sample makes that sample stop being independent, and proposer, verifier and authority stay three separate roles.
Validation methods themselves should be tested against planted weaknesses. The author proposes benchmark models seeded with known defects (temporal leakage, unstable features, rare subgroup failures, calibration defects, hidden interactions, distribution-shift sensitivity) and comparing methods on detection rate, false findings and cost. The author concedes detection rate can't be observed on real models, where only the share of findings that survive confirmation, their business relevance and the effort behind them can be measured.
Volume metrics can improve while model reliability stays flat. The author's hypotheticals: a team shipping five times as many models with no more fit for use, or a validation group finishing twice the reviews while catching fewer material weaknesses. The author proposes measuring development on the cost of reproducible, fit-for-use improvement and validation on confirmed weaknesses that change approval decisions, and predicts a growing queue of models awaiting credible review unless verification infrastructure (reproducible pipelines, executable tests, protected datasets) is designed deliberately.
An agent's arbitrary preference can decide which defensible analysis experts adopt. The author cites an Anthropic study (Herdağdelen, Joren and Tamkin, 2026, "Who's Aligned to Whom?"). In it, 50 causal-inference practitioners evaluated an agricultural advisory program alongside an agent whose methodological preferences had been randomly assigned: difference-in-differences vs ANCOVA, the productivity definition, the estimand, the weighting. The agent argued its side with legitimate evidence and no falsification, and participants followed it 86% of the time, a 72-percentage-point causal effect; even analysts who had already stated a preference became substantially more likely to switch.
The decisive influence comes upstream, before the human's final call. In one case from the paper, an analyst's difference-in-differences estimate of +4.6% met the cooperative's expansion target; the agent pushed a levels-adjusted approach (−1.6%), and the analyst reported −1.3% and recommended discontinuing the program, even rejecting the agent's own final pro-rollout recommendation. The author's point is that this looks like oversight working, but the agent had already shaped the evidence. It is the model-development problem of choices about the target, sample and loss made long before anyone reviews performance.
Sign-off and a convincing explanation do not show independent judgment. In the study, more than half of participants considered the recommendation more their own than the agent's, 77% repeated the agent's arguments (sometimes near-verbatim), and only 15% mentioned the agent. The author concedes that persuasion is not inherently bad, but argues that a validator who accepts the agent's own explanation of its methodology has performed no independent challenge. No bank would call validation independent because it signed a report the development team wrote, so "human in the loop" at the end of a workflow does not meet that standard either.
Separate proposing, verifying and authorizing; another agent's agreement isn't independence. The author's prescription: the LLM proposes several credible specifications, independent mechanisms verify, optimization evaluates the alternatives, and an authorized decision process decides what can be acted upon. Where several approaches stay defensible, the uncertainty should be preserved rather than argued away. The paper reports that an unsteered Claude Opus 4.8 agent chose the same five-choice configuration in 91% of 123 simulations. From this the author infers that development and validation teams using similar models may share analytical judgments despite being organizationally separate, and that AI-assisted replications may produce artificial consensus.
Test the agent's decisions, not just its final answers. Borrowing the study's design, the author proposes holding the evidence constant and systematically varying the agent's analytical preferences. The tests would then check whether it still examines alternatives, which tools and specifications it runs, and whether the final recommendation moves when only its inclination changes. The aim is not identical answers, but to make dependencies explicit and confirm that the decision follows from evidence and authorized criteria.
The OCC’s New AI Framework: Who Is Actually in Control?
Risk follows an agent's authority, not its sophistication. The author endorses the OCC's Supervisory Framework for Bank Use of Super Intelligence (November 2026). It observes that a simple rules-based agent able to initiate transactions can be riskier than a sophisticated model that only drafts internal documents, and proposes classifying by consequences, autonomy and reversibility rather than architecture. The author adds that authority is not static: the same coding assistant becomes far riskier once given repository write access and deployment rights, so approval to use a model must not become approval for every action it later takes.
Validating components cannot establish that the integrated decision process is sound. In the author's credit-underwriting example, the credit model, document extraction and ratio calculations may each pass. Yet the agent can still pull the wrong customer's statements, use an outdated statement, apply the wrong debt definition or call a model outside its approved population. The author splits the work into model validation, orchestration validation (state, tool calls, error propagation, permissions) and decision assurance (evidence, constraints, delegated authority). They call orchestration failures "the risk nobody owns", since MRM, InfoSec, Technology Risk and Compliance can each pass their piece. The business should own the end-to-end process; another AI committee is not the answer.
Human approval isn't a control; separation of authority must be designed in. The author welcomes the OCC's acknowledgement that high volumes, nominal approvals and alert fatigue hollow out human-in-the-loop review: a reviewer of thousands of daily recommendations stays accountable without any realistic ability to catch errors. Their proposal goes beyond the OCC's: the agent proposes, while independent mechanisms trace figures to authoritative records, reproduce calculations and enforce limits at run time. Humans would be reserved for judgment, exceptions and uncertainty, with execution records giving examiners evidence of what was checked and why an action was permitted. The author names this "executable assurance". They note that the OCC document itself was AI-assisted and human-approved, which shows the gap between someone accepting responsibility and anything having been verified.
An LLM judge cannot establish that a decision is correct. The OCC notes the bias, correlated-failure and independence problems of LLM-as-a-judge; the author goes further. A second model approving a credit memo's debt-service coverage ratio and covenant reading shows only that two probabilistic models agree, not that the data, methodology or policy compliance were right. LLMs can help find where more verification is needed, but they should not be the final authority.
Test results need a statistical framework, with discovery kept separate from confirmation. The author's illustration: zero failures in 1,000 tests still gives an approximate 95% upper bound of three failures per thousand, assuming independent, representative tests. That rate could be unacceptable for a high-volume payment system, and ordinary transactions reveal little about rare adversarial conditions. The author proposes three stages: vulnerability discovery as an optimization search (adversarial search, experimental design), independent confirmation of how reproducible and frequent each weakness is, and then a test of whether controls stop it from becoming an unauthorized action. Neither discovery nor representative testing can substitute for the other.
Keeping Weight of Evidence When You Move to Gradient Boosting
Moving to gradient boosting loses the scorecard's shared explanatory language. The author argues SHAP is a valid accounting of model output, but it rests on choices the result doesn't show (background data, how missing characteristics are filled in, log odds versus probability). Feature-level SHAP also splits linked fields such as balance, limit and utilization. So people who can challenge a WoE trend in five minutes struggle to challenge it, and the model looks more like a black box than it is. Their answer, in a new paper with Denis Burakov ("Conditional Weight of Evidence for Gradient-Boosted Credit Models", arXiv:2610.24586), is to extend WoE and IV to GBDTs rather than replace them.
Conditional WoE measures a characteristic's effect given the rest of the file. It keeps the scorecard units and sign convention (bads on top, positive is adverse) but starts from the average for applicants who match on everything else. Worked example: a 10% to 20% default estimate is an odds factor of 2.25, so WoE = log(2.25) = 0.81. Ordinary WoE is the special case where nothing else is known. Omitted characteristics take the mix of values that similar applicants actually have, not zero or missing, and identity-linked fields (balance, limit, utilization) are treated as one group. In a simulation with a deliberately planted interaction, file age above 50% utilization had a true conditional WoE of −0.45 (files under four years) and +0.25 (established files). The method estimated −0.41 and +0.23, while ordinary WoE gave +0.08 and 0.00 and had the right sign only 28% of the time. The author stresses this shows the method can detect the interaction, not that young files are good credit.
Conditional IV gives incremental information directly; coalition IV adds up. Conditional IV can't be negative and is zero only when a characteristic adds nothing, which does in one step what clustering, VIF and stepwise selection do after marginal IV. In the simulation, installment debt fell from a marginal IV of 0.68 to a conditional 0.01, while file age rose from 0.07 to 0.16 because its effect reverses with utilization. In the FICO HELOC data, trade volume fell from 0.24 to 0.04. Coalition IV divides the whole model's IV among characteristic groups: the full file's 2.51 was matched exactly by the coalition IVs, against a 4.56 sum of marginal IVs. That gives a model document a characteristic-level IV table whose rows add to the total.
Two estimators are practical, each with stated weaknesses. The more accurate one fits an extra GBDT per characteristic group to predict the production model's probabilities without that group, and needs no new outcome data. In the simulation it had errors of 0.09–0.18 logit units and the right sign for 99–100% of material cases, but it is only as good as the production model and is untested on a badly misspecified one. Feeding the production model a missing value is not a valid shortcut. The second, closer to scorecard practice, computes bad/good shares within a neighborhood of applicants who often share leaves. It beat ordinary WoE on every characteristic but needs a lot of data, and was sometimes worse than assigning zero for weak characteristics, so the author says to check it against that baseline.
Reason codes depend more on the reference group than on model error. The authors adapt the Regulation B comparison against applicants at or just above the passing score: they swap each characteristic group into reference files and average over swap order and references. The contributions sum exactly to the score gap and reproduce points-below-average on an additive scorecard. A simulated applicant 90 days past due with 53% utilization gets delinquency as the whole reason against broad passers, but revolving credit as the main reason against tree-similar passers, who all have a delinquency and use 27% on average. Switching reference groups changed the leading reason in about a third of simulated declines, while fitted and true models agreed 93% of the time, so the author argues reference choice should be documented as policy. Conditional WoE is reported separately from the reason code. Conceded limits: the accuracy results are simulated, the HELOC cutoff was imposed, booked-only samples bias characteristics the earlier approval policy screened on, and the paper doesn't settle which reference group to use.
A model trained on teacher probabilities converges to the teacher, not the truth. In the author's paper with Denis Burakov (SSRN), exact OLS risk under teacher labels is teacher bias + teacher noise × d/(n−d−1): the noise term shrinks with data but the bias is a floor, so teacher labels win at small n and real outcomes win beyond a crossover (about 19 observations in their worked example). They conclude the optimal weight on teacher labels in a blend goes to zero as data grows. Scoring the student against the teacher cannot reveal the teacher's errors; only realized outcomes can, so they recommend Brier and log loss on outcomes, compared against conventional models fitted to the same outcomes and to the same teacher probabilities.
A frozen pretrained representation pays off only below some sample size. Projecting to r of d dimensions cuts variance but permanently loses signal outside the retained directions ("representation loss"); compression beats the full model only when that loss is below σ²(d−r)/(n−d−1), a threshold that falls toward zero as n grows (crossover near n = 39 in their example). Combined with teacher labels, only the representable part of the teacher's bias appears as bias, while the rest turns into noise-like error in the fitted head — harder to spot, not smaller — and the components don't add neatly (at n = 27, best mixed model risk 0.207 compressed against 0.201 full).
Temperature calibration fixes a score's scale, not its direction. In a Gaussian probit model, a teacher with the right direction but wrong magnitude is exactly repaired by temperature scaling; a teacher pointing in a wrong direction still gets a perfect reliability diagram after recalibration but stays wrong (a 60° rotation: excess Brier 0.086, only cut to 0.063 by exact recalibration). The author's point for validators: a calibrated model can still rank cases poorly, which only discrimination against outcomes or comparison with an outcome-fitted model will flag; under a logistic link temperature scaling is an approximation, but the direction conclusion holds.
Candidate-scoring models must give one event one probability across menus. Softmax enforces summing to one within a menu, not agreement across menus: if "30+ days past due" gets 40% but its split into "30–59" and "60+" sums to 55%, the model is incoherent. With independent per-candidate scoring (Luce's choice axiom, which the author says matches how TypeSafe's Jev describes high-cardinality scoring), coherence holds exactly when a combined event's score is the log-sum-exp of its parts; with shared-context scoring there is no guarantee, so the author says to test the same event under different candidate sets, orders and wordings.
The exact formulas are optimistic for heavy-tailed data at small samples. Rerunning with a multivariate t (5 degrees of freedom), simulated risk was 35% above the formula at n = 12 and about 7% above at n = 95 (uniform inputs came in slightly below); the author notes credit, fraud and AML data are usually heavy-tailed. Stated limits: linear heads on frozen backbones only (not LoRA), a teacher not trained on the student's sample, and population-level calibration results; and a model that predicts which option an annotator or past policy would pick has not been shown to pick the best outcome, which needs a causal model of outcomes given actions.
Dot-product attention is a kernel-weighted average, not something new. Naming the exponentiated score κ(q, kᵢ) = exp(q⊤kᵢ/√d_k) rewrites softmax attention exactly as a normalized kernel smoother; softmax only makes similarities positive and the weights sum to one, and with labels as values it is Nadaraya–Watson (NW) regression (Nadaraya and Watson, 1964). The author's toy case: labels 10, 20, 40 with weights 0.6, 0.3, 0.1 give 16. The argument comes from the author's SSRN tutorial and a theory paper written with Ming Yuan, with a companion notebook that checks each identity.
The weights and the values are two separate choices. The same weights averaged over labels give a prediction, over feature vectors a representation, over residuals a correction (residuals −2, 2, 5 average to −0.1 in the worked example). Changing the similarity rule is a separate step and a modeling change: Gaussian attention with T = 2√d_k equals scaled dot-product attention plus a key-norm bias, so it matches plain dot-product attention only when all keys have equal norm.
Stacked residual attention layers compute iterated NW, i.e. boosting. Context self-attention updates the fit, query cross-attention applies the averaged residual at new inputs, and the skip connection adds it, which is the BoostNW scheme studied by Di Marzio and Taylor; both updates must read the same incoming residual, a bug the author warns is easy to write. In a full transformer each head is still a kernel smoother, but the output projection, layer norm and token-wise feed-forward network mean the output is no longer an average of the original features. The construction shows attention can implement iterated NW, not that a trained transformer does.
Initializing a trainable network as NW beat random initialization. On a four-layer restricted residual model (not the full block), test RMSE over 128 episodes was 0.290 after 250 updates and 0.263 after 1,000 with NW initialization, against 0.548 and 0.394 for PyTorch's default and 0.380 for untrained iterated NW, across three seeds. The author concedes the runs had not converged, the test functions came from the training family so nothing is shown about transfer, and shuffling labels (RMSE about 1.0) shows the network uses label–input pairing without explaining why.
The kernel reading has limits, and validation must set depth. Context error keeps falling with layers while validation error levels off and rises, so depth acts like early stopping; a positive, row-stochastic attention matrix with eigenvalues 1 and −0.8 makes residual iteration diverge (×1.8 per step) at any step size; and iterated NW converges to an interpolating fit that still differs from ridgeless kernel ridge regression (KRR) at new inputs. Worked examples, not benchmarks: on the synthetic data NW scored 0.241, iterated NW 0.205, KRR 0.209; on a California Housing split iterated NW won on validation yet lost on test (1.105 vs 0.756 for NW and 0.632 for KRR), which the author keeps as the case for a separate test set.
When AI Becomes an Insider, Who Is Left to Check Its Work?
Insider-risk controls cannot tell whether an authorized decision is correct. The author accepts Satya Nadella's essay "Models as Insider Risks in the Super Intelligence Era" (restrict privileges, monitor, keep the ability to intervene, separate a model's intelligence from authority to act), but argues security and decision correctness are different problems. In the author's hypothetical, a credit-portfolio agent acts entirely within its permissions with a full audit trail yet recommends a reallocation that breaches concentration tolerance, perhaps through a misread constraint, a bad correlation assumption or ignoring stressed losses; an audit trail records what it did, not whether it should have, and a kill switch helps only if something notices.
Verification must be engineered, not handed to another LLM. The author's proposal: the LLM proposes, but acceptance criteria sit outside its control and are checked by calculation, authoritative records, validated statistical models and optimization (for feasibility and better alternatives); "there is no reason to ask another LLM whether a numerical constraint has been satisfied when we can calculate it directly." Where Nadella stresses transparent reasoning, the author prefers independently reproducible evidence, since a chain of thought may not faithfully explain an answer. The concession: the checks themselves can faithfully implement the wrong constraint, so they need testing by people who understand the maths, data and business.
Automating junior work erodes the expertise that independent validation needs. Drawing on their Detroit engineering and model-validation career, the author argues judgment is built by doing the work, and validators need technical competence, not just a separate reporting line. The predicted path: fewer junior hires, survivors reviewing AI output instead of building analyses, seniors retiring, until validation becomes procedural; the author imagines AI validating an AI-built model, writing the report, and a committee approving it from AI summaries. The risk is not that AI work is wrong but that firms lose the ability to tell excellent work from "plausible nonsense".
Productivity gains are measurable; expertise erosion is not, so it compounds. Firms consume expertise built under the old system without replenishing it, and as verification gets harder they automate it further, a reinforcing cycle of dependence the author says is not inevitable, since AI could instead speed up learning if work is organized deliberately. The illustration: an analyst spending three days on a model's odd behaviour looks less productive than one getting a polished AI explanation in three minutes, yet learns more. The author sees the same risk scaled up as a "governance paradox": ever more policies, dashboards and AI monitoring AI around systems fewer people understand.
The author prescribes four practical responses to keep institutional competence. (1) Separate generation of candidate actions from evaluation and authorization, with criteria the proposing AI cannot change, and reserve humans for judgment, unresolved uncertainty or out-of-limit cases, since blanket human approval does not scale. (2) Redesign junior roles to build judgment: use AI to explore alternatives but still understand the calculations, reproduce results independently and investigate disagreements with established methods, with real mentoring. (3) Keep in-house technical capability to challenge vendor and foundation-model systems, because relying on vendors to vouch for reliability "outsources" the ability to judge. (4) Measure competence alongside productivity through periodic challenge exercises: unfamiliar problems, planted errors, reproduction by alternative methods.
Learning the Geometry Before Fitting the Regression
Split tabular prediction into a learned comparison rule and a linear solve. The author and Aijun Zhang (paper: "Learning Spectral Geometry for Kernel Regression", SSRN) learn a kernel that sets how rows are compared, then fit a regularized kernel ridge regression given that kernel; a tree ensemble fits both at once, and its splits mean nothing on a table with a different schema. The stated motive is reuse: learn the geometry once and need only a linear solve on a new task. They concede the paper tests this only within single datasets, and cross-table transfer (which needs a schema-agnostic representation and a cross-task objective) is still to be shown.
Each learned component handles scale, direction or decay of similarity. Each numerical input becomes K sine-cosine pairs with trained, ordered frequencies (drawn from Bochner's theorem, but as a learned representation rather than random or spectral-mixture features). A positive-definite metric B weights the pairs, and its cross-feature terms let joint changes (location with income) count differently and make the kernel nonstationary. A Laplace kernel with learned bandwidth T was chosen over the Gaussian for three reasons: it is positive semidefinite in any dimension (Schoenberg), it suits irregular response surfaces, and it is nonadditive, so interactions come without specified interaction terms.
A canonical parameterization makes the fitted geometry readable and comparable. Free amplitudes, metric and bandwidth are redundant: rescalings leave every prediction unchanged, so feature weights have no defined value. Fixing the metric to unit trace via a normalized matrix exponential removes this, and pair and feature "masses" can then be read off. In the California Housing model, latitude takes about 45% of the trace and longitude 31%, put mostly on their highest frequency, while income and house age sit on low frequencies. Sensitivity scales with frequency squared, so location is resolved finely and income is treated as slowly varying.
On California Housing, the exact ridge solve mattered as much as the geometry. Geometry is trained episodically (fit ridge on a support set, score on a disjoint query set, backpropagate through the solve), then a final exact ridge fit is done on all training rows. Across three 80/20 splits the refined model's RMSE is $820 below tuned CatBoost (1.90%), lower on every split ($1,107, $1,001, $351). Swapping a 2,048-landmark Nyström approximation for the exact solve cut about $1,700, roughly two fifths of the total gain; refining the geometry cut about $2,372 more and full-pool refitting about $78.
Results on mixed and binary data are mixed, with CatBoost winning Adult. Categorical fields enter as fixed orthonormal contrasts under one shared metric, and binary scores are sigmoid-calibrated, as are CatBoost's. The spectral model has lower RMSE on Bike Sharing on all three runs and AUC higher by 0.003–0.006 on Taiwan with log loss within 0.0011, but CatBoost is more accurate on every Adult measure. Code, checkpoints and predictions are at github.com/asudjianto-xml/Tabular-Spectral-Kernel (v0.1.0), with separate verify, replay and full-training reproduction levels.
I Have Seen This Movie Before. It Was Called Cost Cutting in Detroit.
Cost-cutting pays its rewards before its consequences arrive. From the author's years as a Detroit automotive engineer: savings were immediate and booked, while warranty claims, failures and recalls surfaced two or three years later under a different executive, whose response was often another cost-reduction drive. The accounting can be correct (a $5 cut on a component across a million vehicles is legitimately $5 million saved) while the economics are wrong, because the cost may only have been moved into the future; short evaluation periods and management rotation leave the liability with the organization and not the decision-maker.
You can outsource the work but not the responsibility. The author's engine team outsourced the cylinder-head gasket, lost the in-house skill to design it, and when gaskets leaked could no longer diagnose what is really a system problem (thermal expansion, clamping force, surface finish) or challenge the supplier's fix. They saw the same weakness in the 2008 credit card default wave: capabilities that looked redundant in stable years turned out to be the capacity to handle unfamiliar conditions, and expertise "cannot be hired, trained or reconstructed overnight". The line they draw is between outsourcing execution and surrendering the ability to understand it.
AI productivity metrics count the output and miss the judgment. If a ten-hour analysis now takes ten minutes, the author asks what the ten hours held: noticing data inconsistencies, questioning assumptions, deciding the question was wrong, none of which appears in the productivity calculation. Their predicted loop: juniors who mainly approve AI output (analysts, software engineers) don't build the expertise, success justifies further staffing cuts, and remaining staff become less able to evaluate the AI, increasing reliance on it. They concede this is not inevitable if AI is deliberately used to build expertise.
Two models agreeing is not independent verification. The author applies the engineering distinction between designing something and demonstrating it meets requirements: execute code to check a calculation, check claims against an authoritative database, test predictions against independent observations, evaluate business or regulatory constraints explicitly. An organization that has lost its experts and lets AI judge AI risks "automating not only the work but also the illusion that the work has been properly checked", and unlike a leaking gasket, a flawed analysis may give no physical signal for years.
Spare human expertise is a safety margin, not inefficiency. The author forecasts the Detroit cycle repeating at larger scale: an AI transformation is declared a success, executives move on, expertise erodes, then a downturn, technology failure or regulatory exam exposes decisions nobody can explain, and remediation consumes the savings. Their prescription is to retain enough people who understand the business, data, models and systems to challenge the technology, especially in a crisis when historical precedent fails. They allow that some firms are investing in verification and workforce development, and say they do not oppose automation, only careless choices about what gets automated away.
Validation is already agentic; give it two loops, not ReAct. The author argues that using generative AI to summarize documents, read code and draft reports only speeds up existing tasks, while a good validator already runs an adaptive investigation in which each result sets the next question. The author proposes giving this a computational form as two loops. The inner loop does nearly all the work with no language model: build a test from a versioned specification, run the model under validation, score against a reference, update the score table and Pareto frontier, and let a numerical optimizer pick the next test. The outer loop runs rarely: when the test space seems unable to express a failure, a language model, domain expert or rule proposes a new factor, which software checks for buildability and scorability before versioning the specification. The author contrasts this with ReAct-style "prompt and pray".
Every validation activity is a discovery loop with its own evaluator. What changes across data validation, implementation verification, performance validation, benchmarking and monitoring is the evaluator and the objective. The loop structure stays the same. The author's worked example is an XGBoost 12-month probability-of-default model: a utilization/account-age relationship shifts across vintages, an optimizer searches threshold combinations under a minimum-support rule (e.g. accounts under 12 months with utilization above 70%, which the author labels illustrative), and the model, membership rule and comparison are fixed before reserved vintages are checked. Loops hand off to each other, so a vintage-concentrated effect triggers a data check, then benchmarking. A validation knowledge graph should record hypotheses, tests, results and open questions so that software neither repeats dead ends nor treats a conjecture as a finding.
Search for conditional weakness, then confirm on withheld evidence. The target is poor performance in particular conditions that averages hide, such as an AML model failing for one small business type or an agent misbehaving when one tool fails. Tests cross a candidate array (customer regions, or task presentations such as a conflicting instruction) with a condition array (scenarios, vintages, tool faults), each tied to a reference fixed before search begins. In the author's hypothetical, region A shows 20%/80% loss increases (ordinary/stress) and region B 60%/50%: averaging ranks B worse (55% vs 50%) and hides A's stress failure, so both stay on the Pareto frontier. Because discovery scores carry favorable noise, findings are confirmed on reserved data under a pre-declared acceptance rule with a multiple-testing correction, failures stay in the record, and report claims are checked against source-linked values (a recorded 0.058 supports "0.058" and contradicts "0.100").
The test is what still runs without the language model. A ReAct agent with its model removed does nothing. The discovery loop still builds, runs, scores and selects tests, and loses only the ability to propose new kinds of test. The author argues ReAct fits validation poorly because proposal, execution and judgment sit in one model that often scores itself, its memory is a free-text trace, and nothing separates discovery evidence from confirmation evidence. The language model's real contribution is reaching outside the current representation: no search over single-prompt settings can place a conflicting instruction several turns before the question, but a model can propose a conversational test family with that separation as a factor. The author's condition: a stalled search is not enough reason to ask for one, since small budgets, poor starts, noise or a flat objective should be ruled out first.
Controls sit at decisions, and authority and materiality stay human. Scope limits the questions, permissions limit the tests, verification limits the findings and designated authority makes the decisions. A human reviewer in the workflow does not enforce anything on its own. The proposer, the verifier and the decision-maker should be separate, and a second check counts only if it brings evidence beyond the first component's judgment, since two LLMs can share a misreading. Verification should use the least inferential instrument that can answer: retrieve, then calculate, then simulate with stated assumptions, with semantic judgment kept for interpretation. Because hypotheses now arrive faster than they can be verified, the validator's job shifts to directing the search. That means feeding it context (intended use, assumptions, prior findings), asking whether the search space and evaluator can capture the relevant failure, letting the system contradict them, and owning the materiality judgment rather than just accepting or rejecting a software recommendation.
Import banking MRM's principles, not the machinery that grew around them. He keeps four principles: independent challenge, evidence that survives challenge, explicit acceptance of residual risk, and ongoing challenge after approval. What he would leave behind accumulated "one remediation at a time": inventory treated as the control, documents treated as evidence, independence meaning re-performing the developer's work, calendar-driven validation, and headcount growing with model count.
Govern the whole decision system, including its human reviewers. He separates small-m (the model, y = m(x)) from Big-M (data, models, optimizer, rules, humans and action), arguing every component can be green while Big-M fails in the gaps between functions; human-in-the-loop is part of Big-M and should be tested together with the AI. His proposed operating model has the business own the decision, one shared intake and tiering across all risk functions with joint review for the top tier, MRM leading an integrated cross-boundary challenge, and humans setting an "envelope" of limits that the architecture enforces on every action, with exceptions escalated.
Tier the use, not the foundation model. One LLM can sit behind memo drafting and client-facing account answers alike, so the tier comes from five dimensions of the use: impact, autonomy, authority, uncertainty and scale. Tiers should be inherited by low-tier parts feeding high-tier decisions, should add up when low-risk tools combine into one agent, and should expire on new permissions or channels; the most common failure he has seen is a read-only internal pilot gaining write access and a client channel six months later without re-tiering.
Evidence, not documents, must be what validation produces. Because AI-generated documents are cheap and say little about the work behind them, he wants inventories to store replayable chains of test, configuration, data, result, verifier, finding and decision, with the report as one generated view; the inventory becomes a map of which models, prompts, data and tools feed which decisions. His scalability test: if headcount scales with model count the design fails, so validation should be change-triggered, automated and replayable, with exceptions escalated by risk.
Validation should be machine-run discovery confirmed with statistical rigor. In an illustrative case, 86% overall accuracy hid 96% under most conditions and 45% under adversarial framing; a space-filling design turned 20 labeled complaint cases into 120 scenarios, and a knowledge-graph answer key scored a correct overdraft fee at distance 0.056 versus about 1.55 for a fabricated $50 fee. He insists on separating discovery from held-back confirmation and correcting for multiple findings — in his Benjamini–Hochberg example a raw p of 0.049 is retained against a 0.040 cutoff — and concedes a single system coordinating a full validation is still being built.
Perhaps Agentic AI Needs a Little Old-Fashioned Process Engineering
Making the LLM the universal orchestrator is the real hammer-and-nail error. Agents already get rich toolboxes, but the LLM decides which tool to call, interprets results, judges adequacy and ends the sequence, so orchestration is non-deterministic even when tools are not. He argues the pattern's ease of building — short code, impressive demos — is part of the problem.
Start from the process, framed as a sequence of decisions. Borrowing from process re-engineering and Six Sigma, he frames knowledge work such as an AML investigation as State → Decision → Action → New State, then asks how each decision should be made. Often the answer is a database lookup, calculation, statistical test, model, optimizer or rule, and steps that must always happen (a risk calculation before an action) should come from process design rather than an LLM's choice.
The LLM belongs in an outer loop that expands the search space. An inner loop generates, verifies, scores and gates candidates within the known space with criteria outside the LLM; only when it stalls does the LLM, given the verified history, propose new candidates (Aₜ₊₁ = Aₜ ∪ ΔAₜ) that must pass the same checks. In his AML example the LLM might suggest examining payment timing around an unrelated inflow — a hypothesis then tested by code, graph analysis and statistics — but it may never conclude the customer is suspicious.
Remove authority by architecture rather than adding guardrails. His non-negotiable rule is that anything from an LLM enters as a hypothesis, never a fact, and the LLM simply lacks authority to make final decisions, which beats patching an open-ended path space with rules. A proposer is allowed to be wrong; reliability is a system-level target that assumes it sometimes will be.
Process-engineered agents are harder and need more than software engineering. Someone must specify states, decisions, objectives, evaluators, constraints, gates and when the inner loop has stalled, drawing on statistics, experimental design, ML, optimization and domain knowledge. He argues cheap generation makes verification more valuable, complexity is relocated rather than removed, and a knowledge graph should hold the persistent process state rather than the context window.
A healthy model can sit inside a failing decision system. He distinguishes small m, the mathematical model, from big M, the whole chain of data, cutoffs, policy rules, pricing, overrides and people that turns a probability of default into a customer decision. Drawing on engine design at Ford, he argues a car can misbehave with every component in spec because the failure lives in the interaction — "the gap."
Risk is organized vertically while decisions run horizontally. MRM, Data Governance, Technology Risk, Cyber, Compliance and the business can each sign off and every box still be green while the system is wrong. AI sharpens this because behavior comes from the assembly of model, prompts, retrieval, memory, tools and permissions, and the distance from bad output to real consequence is shrinking.
The human in the loop is a component, not a control. A reviewer facing hundreds of AI recommendations a day may lack evidence or time, and after fifty correct answers may just click Approve or be talked into overrides by a persuasive explanation. He wants the AI-plus-human system tested, not the AI with a human bolted on.
Validation must expand from components to system behavior. He rejects having MRM absorb the other functions — one giant box recreates the problem — and also rejects a bigger meeting of specialists each validating their piece; someone must own the interactions. That requires validators who connect business, quantitative, ML, technology and systems knowledge rather than merely representing them.
Governance should be built into the architecture. Whatever can be checked independently — database facts, rules, calculations, simulations — should be, with explicit gates rather than model confidence authorizing actions and humans given real evidence instead of "a persuasive paragraph and two buttons." He likens letting one AI act as investigator, evidence, judge and jury to asking the defendant to deliver the verdict.
Written policy for an agent is not a control. Unlike a trader, an agent has no fear of losing a bonus, job or reputation, so nothing restrains a strategy that merely works; he contrasts writing a $100 million position limit in a handbook with configuring the trading system to reject the next dollar. Adding an LLM judge to read the policy "has added another opinion," not an enforceable constraint.
Harmful behavior can be composed entirely from permitted actions. We cannot list in advance every bad strategy a capable agent may find, and coding agents can compose permitted capabilities into new programs; interacting agents could discover coordination resembling collusion without intending it. Sandboxing and scoped credentials are basic hygiene but control capabilities, not the behavior built from them over time.
Agent count is a meaningless KPI; capability and authority matter. He imagines a CEO boasting of going from 500 traders to 100,000, and notes a document-summarizing agent and one that can read customer data, run code and initiate transactions each count as one. The useful figures are how many can execute code, write to production, move money, communicate externally, spawn agents or run without review.
Autonomy in thinking should not mean authority in acting. He wants broad autonomy for search, ideas and experiments but argues that, because constraints cannot all be specified, governance must engineer the whole environment — observability, delegation, interaction monitoring and escalation when uncertainty rises. His warning is that enterprises may be industrializing agents before industrializing agent engineering.
Agentic AI handed authority to the LLM first, then bolted on guardrails. He traces how ReAct (Yao et al., 2022), a research pattern tested on question answering and WebShop, became a working definition of "agent" — letting the LLM decide what happens next — before tools gained write permissions and governance became an afterthought. He cites the 2025 AI Agent Index, covering 30 agents, as finding developers disclosed far more about capabilities than safety evaluations.
Separation of duties means separate authority, not more LLM calls. A bank officer proposing a $500 million credit does not also validate it, waive limits and wire the money; chaining LLMs to propose, critique, judge and approve creates "a committee of autocomplete." LLM judges are fine as soft quality signals but not as the source of authorization.
ProveAct confines the LLM to proposing, with the system deciding. Outside the generative loop, the system runs Verify → Establish Feasible Set → Optimize → Gate → Act. In his $500 million credit example the LLM suggests structures (e.g., $350 million secured), the system adds its own candidates including declining, approved models verify, limit-violating options are simply infeasible, an optimizer picks among the rest, and a gate checks delegated authority before execution.
CaMeL applies the same instinct to prompt injection. Debenedetti et al. (2025) separate control and data flows and use capability-based restrictions rather than trusting the LLM to spot malicious instructions, after AgentDojo showed injections could redirect agents in email, banking and travel tasks. Both rest on not making a guarantee depend on the probabilistic model behaving.
ProveAct only works where claims can be verified. He concedes it suits credit, pricing, portfolio, code deployment and fraud work, not open-ended tasks like ten-year corporate strategy, and that a biased generator can omit good options. His mitigations are treating generation as search and having the system inject baselines, defaults and a no-action option, so that a wrong LLM proposal is just one that fails verification.
Most agent architectures rest on prompting and hoping, not mechanisms. The author coins "Prompt and Pray" for designs, like a hypothetical bank agent handling a $10,000 credit-line increase, where LLM interpreter, planner, orchestrator, policy reader, eligibility reasoner and judge all depend on the model getting it right. The author's objection is not to prompting itself but to using a prompt where a mechanism could have been built.
Use an LLM judge only after lookup, computation and prediction fail. Many judged questions have correct answers: whether a document reports a capital ratio can be looked up, whether 17 transactions sum to $4.8 million can be added, a limit breach is a numeric comparison, and default risk is what a credit model estimates. The author applies the same order (lookup, compute, predict or simulate, then LLM) to every component, arguing that moving toward LLM discretion raises the LLM's latitude, not the system's intelligence, and is justified only by real ambiguity, such as "I need some additional room on my card" or open-ended research.
A prompted step is an instruction; a state machine is a control. For a credit workflow (retrieve account, verify identity, retrieve policy, calculate exposure, score risk, check limits, authorize, execute), an LLM orchestrator told "you must always perform these steps" usually complies but cannot be forced, whereas a state machine makes reaching execution impossible until prior states complete. Likewise, two LLMs agreeing an action meets an approval threshold is not knowing it does; a settled rule should be encoded and enforced, with LLM interpretation kept for policy language that is genuinely ambiguous.
Grant LLM freedom early, tighten the architecture toward consequential action. The author proposes a propose → verify → optimize → gate → act sequence, in which the LLM does most of the proposing, verification looks up facts, computes and uses validated models, optimization runs against explicit constraints, and the gate checks policy, permissions, limits and approvals, so uncertainty shrinks at each stage rather than piling up. The author calls the target "Propose and Prove", and concedes that much cannot be proven and belongs to judgment, LLM judgment included, so long as verification is not skipped merely because it is harder to build.
Prompt and Pray persists through vendor incentives and engineering convenience. The author argues providers favour LLM-heavy designs because each planner, critic and judge multiplies inference calls they sell, and that real verifiers need schemas, data contracts, SQL, calibration, rules, permissions, provenance and tests, against thirty seconds to type a judge prompt. The author's review test for every component is "why is an LLM doing this?": ambiguity, an unenumerable search space, synthesis or option generation are good answers, while "it was easy to prompt" is not.