NumTrace: Auditing the Number Provenance of Financial Analyst Agents

Tianlei Zhu, Matt White, Xue Liu, Xiao-Yang LiuarXiv 2026 · arXiv

TL;DR

AI financial agents are surprisingly good at copying digits but struggle to understand what those digits actually represent. NumTrace is a new benchmark that audits the "provenance" of numerical claims, revealing that the vast majority of AI errors are "Right Number, Wrong Account" (RNWA) mistakes—where a model correctly retrieves a number from a filing but misattributes it to the wrong company, time period, or unit.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly deployed as autonomous financial analyst agents, their reliability is paramount. However, existing benchmarks often focus on whether the final answer is correct, failing to diagnose why an agent failed.

In the financial domain, a number’s meaning is defined by its context: the entity it belongs to, the accounting concept it represents, the time period it covers, and the units used. A model might report a "correct" number found in a document, but if that number refers to a subsidiary’s debt rather than the parent company’s, the answer is fundamentally wrong. The authors identify this as the Right Number, Wrong Account (RNWA) error. Current evaluation methods that only check for the presence of a number in a source (number-in-source) or compare against a final reference value cannot distinguish between a simple typo and a deep semantic misunderstanding.

Methods

The authors introduce NumTrace, a benchmark designed to audit the source and financial semantics of every numerical claim an agent makes.

The core of NumTrace is a three-verdict auditing system:

  1. Value Check: Do the digits and arithmetic match the cited source?
  2. Binding Check: Does the number refer to the specific entity, concept, period, unit, and scope that the agent claims?
  3. Support Check: Does the cited source actually back both the value and the binding?
Figure 2: Overview of the NumTrace benchmark. (a) On a frozen snapshot of SEC filings, we build the questions and reference answers according to accounting rules. (b) The agent under test gets the question and searches the snapshot; it never sees the reference answers. (c) Each reported number gets a provenance record and three checks; a number that passes the value check but fails the binding check is a right number, wrong account (RNWA) error.

To evaluate agents, the researchers developed a standardized four-stage pipeline: Retrieve (using BM25), Extract (quoting lines), Calculate (applying formulas/scales), and Synthesize (writing the final report). This allows them to pinpoint exactly where in the reasoning chain an error first appears.

Datasets

NumTrace utilizes a frozen snapshot of 2,163 SEC EDGAR documents (10-K, 10-Q, Form 13F, and proxy statements) covering 30 large U.S. companies and 30 institutional managers.

The benchmark consists of 540 questions and 2,306 answer slots. Notably, it is the first benchmark to tackle:

  • Form 13F: Complex reports on institutional share holdings.
  • Item 402(i): Detailed tables regarding executive deferred compensation.

Reference answers were generated using 216 deterministic Python rules based on actual SEC instructions and accounting standards, ensuring a "ground truth" that accounts for nuances like unit changes over time or co-filing group memberships.

Key Insights and Ideas

  • The RNWA Trap: The most common failure isn't "hallucinating" digits out of thin air; it's "hallucinating" the context. An agent might see the number "256" in a filing and report it as a group's total holding, when it actually only represents one small member of that group.
Figure 1: An RNWA error: GPT-6 Astra reports one firm’s 256 shares as the Point72 group’s holding (refer- ence value: 2,854,645). Figure 6 gives the full record.
  • Audit the Provenance, Not Just the Answer: By requiring a "provenance record" for every number, NumTrace forces agents to show their work. This makes the AI's reasoning "certifiable"—a human or another system can verify if the evidence cited actually supports the claim made.
  • Complexity of Financial Semantics: Financial documents are "traps" for LLMs. For example, Form 13F changed its reporting units from thousands of dollars to single dollars in early 2023. Agents that don't account for this date-dependent rule will consistently fail the binding check on units, even if they copy the digits perfectly.

Key Results

The authors audited three state-of-the-art agents: Claude Opus 5.5, GPT-6 Astra, and DeepSeek-V4.1-Flash.

  • RNWA Dominance: RNWA errors accounted for 67.5% to 81.8% of all failing claims across the three agents.
  • High Fidelity, Low Accuracy: While agents passed the Value Check for over 91% of claims (meaning they are excellent at copying), their Binding Check pass rates were significantly lower, particularly on 13F filings (ranging from 65.4% to 82.7%).
  • Group Totals are the Hardest: On questions involving co-filing groups (13F-GROUP), binding pass rates plummeted to between 41.9% and 48.5%. Agents frequently reported a single filer's position as the total for the entire group.
Figure 4: Agents rarely miscopy numbers, but when asked for a co-filing group total, more than half of their scored claims pass the value check but fail the binding check. Each bar splits one agent’s scored claims; co- filing group totals are a subset of 13F.

Limitations and Future Work

  • Retrieval Bottlenecks: The authors noted that missing evidence in the retrieval stage accounted for a portion of the 13F errors, suggesting that better search mechanisms are needed alongside better reasoning.
  • Scorer Sensitivity: While the automatic scorer is highly consistent, human expert review found that it still has a false-fail rate of about 20% for binding, often due to the nuances of natural language synthesis.
  • Toward Runtime Certification: The authors suggest that the next step is developing "runtime certifiers"—systems that prevent an agent from releasing a numerical claim unless its provenance record can be independently verified against filing rules at the moment of generation.

Summary updated

Abstract

A numerical claim generated by a financial analyst agent can match a number in a company filing but refer to the wrong entity, concept, period, unit or scope. A check based on reference answers can mark such an answer as wrong but does not explain why it fails. In this paper, we present NumTrace, a new benchmark that audits the provenance of each numerical claim made by financial analyst agents: its source and financial semantics. NumTrace currently has 540 questions over 2,163 SEC EDGAR documents. To the best of our knowledge, it is the first numerical benchmark about who holds a company's shares (Form 13F) and what its executives are owed (Item 402(i)). On a frozen snapshot of these documents, we build the reference answers according to accounting rules. NumTrace then gives each claim three separate verdicts: value, binding and support. By retaining separate value and binding verdicts, NumTrace detects right number, wrong account (RNWA) errors: a claim has an RNWA error when it passes the value check but fails the binding check. The RNWA rate quantifies the gap between value-only and joint value-and-binding pass rates. Finally, we audit three agents, Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash. RNWA errors make up 67.5-81.8% of each agent's failing claims. In particular, on 13F co-filing group questions, about 91% of claims pass the value check, but fewer than half pass the binding check.
BibTeX
@misc{zhu2026numtrace,
  author = {Tianlei Zhu and Matt White and Xue Liu and Xiao-Yang Liu},
  title = {{NumTrace: Auditing the Number Provenance of Financial Analyst Agents}},
  year = {2026},
  eprint = {2610.34887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34887},
}
Figure