Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

Cosimo Galeone, Anna Ettorre, Ginevra Mariani, Minsu Park, Giuseppe Ettorre, Daniele LigorioarXiv 2026 · arXiv

TL;DR

This paper moves beyond static leaderboard scores for financial AI agents by introducing a rigorous, auditable framework for measuring output drift, diagnosing failure ownership, and validating interventions. Through a case study on a "shipped" agent, the authors demonstrate that while automated optimization loops can reliably slash operational costs by ~66%, many apparent accuracy gains are indistinguishable from inherent run-to-run noise.

What Problem Does This Paper Address?

In the high-stakes world of finance, a single leaderboard score is insufficient for production-grade AI. If a financial research agent provides a wrong answer, a compliance officer needs to know more than just the error rate; they need to know why it failed and who in the organization (the prompt engineer, the tool developer, or the model provider) is responsible for the fix.

Current benchmarks often ignore output drift—the fact that the same agent, given the same question, might pass or fail on different runs due to LLM stochasticity. Furthermore, they lack a standardized failure taxonomy that maps errors to actionable remedies. This paper seeks to build an "audit trail" for trust, moving from "what is the score?" to "how stable is the system, and can we reliably improve it?"

Methods

The authors conducted a controlled case study using a reference agent from the Vals Finance Agent Benchmark v2. Their approach relies on a strictly "frozen" protocol to prevent p-hacking:

  1. Drift Measurement: They ran the identical agent configuration seven times on a development set to establish a "noise band" for both accuracy and cost.
  2. Failure Taxonomy: They developed a taxonomy based on remedy ownership:
  3. Policy: Errors in the agent's instructions (owned by prompt engineers).
  4. Tool: Issues with interfaces or output parsing (owned by platform engineers).
  5. Data: Failures in the retrieval pipeline (owned by data owners).
  6. Model: Inherent reasoning failures (owned by model risk teams).
  7. Automated Optimization Loop: Using the AloLab engine, they implemented a loop where an "Analyzer" (Claude Sonnet 5) reads failed traces and proposes targeted changes. These changes are only kept if they pass a pre-defined statistical gate on a development split.
  8. Auditable Protocol: To ensure trust, they froze the test split, the decision rules, and the failure taxonomy before any test trajectories were analyzed.
Figure 1: The system and the protocol. Agent, judge, analyzer and loop form the optimization cycle of §6, run on the dev split only. Shaded boxes are frozen before use: the test split and its decision rule before any test number existed (§6.3), the taxonomy before the test trajectories were analysed (§4.1).

Datasets

  • Vals Finance Agent Benchmark v2 (Vals FABv2): The primary source for questions and expert-written rubric criteria.
  • Dev/Test Split: A stratified 18/9 split of the public Vals FABv2 questions, ensuring issuer-disjoint sets to test generalization.
  • Agent Trajectory Interchange Format (ATIF): Used for standardized exporting of agent traces for analysis.

Key Insights and Ideas

  • The Score is a Draw: One of the most striking findings is that a "leaderboard score" is just a single sample from a wide distribution. Identical runs of the same agent saw pass rates fluctuate by 12 percentage points and costs vary by 17%. Without measuring this drift, researchers risk over-interpreting minor "improvements."
  • Diagnosis is Corroborated: The authors used "blind readers" (different LLM families) to diagnose failures without seeing the Analyzer's output. They found 94% agreement on what failed, suggesting that LLM-based failure diagnosis is highly reliable for identifying the fault, even if pinpointing the exact location in a long trace remains difficult.
  • Policy is the Primary Failure Mode: For this specific shipped agent, 65% of failures (13 out of 20) were attributed to "Policy." This suggests that even with state-of-the-art models, the way we instruct agents remains the most significant bottleneck and the most frequent source of error.
  • Cost Savings are "Real," Accuracy Gains are Often Noise: The optimization loop successfully identified a structural change (switching a nested retrieval model to a cheaper variant) that yielded massive cost savings. However, a 14-point accuracy gain on the development set evaporated on the test set, falling within the previously measured noise band.
Figure 2: The four measurements. Top left on the dev split, the other three on the frozen test split. (a) Cells passed by the shipped configuration in seven identical dev runs. (b) Primary layer of the 20 failed test trajectories per baseline replica. (c, d) Both loop winners on the frozen test, three replicas each, paired bootstrap over the nine questions: the cost winner’s cost interval lies entirely below the drift band; every accuracy interval includes zero.

Key Results

  • Cost Reduction: The optimized "cost winner" configuration reduced expenses to 0.34x of the baseline (95% CI [0.22, 0.58]) on held-out questions, while maintaining accuracy within the noise floor.
  • Accuracy Stability: While the "accuracy winner" gained 14 points on dev, it only gained 1.5 points on the test set (CI [−5.3, +7.2]). Because this interval includes zero and stays within the drift band, the authors correctly identified it as a non-significant result.
  • Failure Anatomy: On the test set, failures were owned by Policy (13), Tools (3), and Model (4). Notably, zero failures were attributed to "Data" under their specific projection, highlighting that "stale filing" errors were actually failures of the agent's policy to check dates correctly.

Limitations and Future Work

  • Case Study Scope: The authors emphasize this is a study of one agent and a small set of test questions (n=9). The results are an "anatomy" of failure rather than a population estimate.
  • Human-in-the-loop: While model-based "blind readers" corroborated the diagnosis, the question of whether human financial analysts would agree on "remedy ownership" remains for future research.
  • Causal Efficacy: The study validates the end-to-end optimization loop but does not isolate the specific causal impact of the diagnosis itself versus the loop's search strategy.
  • Future Directions: The authors suggest that practitioners should adopt this "audit trail" approach: freeze splits, run multiple baselines to measure drift, and write promotion rules before looking at test results to ensure that "trust" is built on evidence rather than noise.

Summary updated

Abstract

Can you trust a finance agent you did not build? We take the reference agent shipped with the Vals Finance Agent Benchmark v2, leave it untuned, and ask four questions a leaderboard score does not answer. Is the score stable? Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%. When the agent fails, who owns the fix? We classify 20 failed trajectories, from three runs of nine held-out questions, by the layer whose owner would remedy them, under a taxonomy frozen in advance: 13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data. Can that diagnosis be trusted? Four blind readers from three model families, given only the traces, name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%). Can a change be trusted? An optimization loop reads the traces, makes one targeted change per round and keeps it only if it clears a fixed threshold. Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise. Under an accuracy objective it gains 14 points on the development questions and 1.5 on the held-out ones (CI [-5.3, +7.2]). Judged against the measured run-to-run noise, under a rule written before the test was opened, the first is a real saving and the second is not. This is a case study of one agent and nine test questions, not a general result.
BibTeX
@misc{galeone2026can,
  author = {Cosimo Galeone and Anna Ettorre and Ginevra Mariani and Minsu Park and Giuseppe Ettorre and Daniele Ligorio},
  title = {{Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent}},
  year = {2026},
  eprint = {2611.17939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17939},
}
Figure