TL;DR
AI financial agents are surprisingly good at copying digits but struggle to understand what those digits actually represent. NumTrace is a new benchmark that audits the "provenance" of numerical claims, revealing that the vast majority of AI errors are "Right Number, Wrong Account" (RNWA) mistakes—where a model correctly retrieves a number from a filing but misattributes it to the wrong company, time period, or unit.
What Problem Does This Paper Address?
As Large Language Models (LLMs) are increasingly deployed as autonomous financial analyst agents, their reliability is paramount. However, existing benchmarks often focus on whether the final answer is correct, failing to diagnose why an agent failed.
In the financial domain, a number’s meaning is defined by its context: the entity it belongs to, the accounting concept it represents, the time period it covers, and the units used. A model might report a "correct" number found in a document, but if that number refers to a subsidiary’s debt rather than the parent company’s, the answer is fundamentally wrong. The authors identify this as the Right Number, Wrong Account (RNWA) error. Current evaluation methods that only check for the presence of a number in a source (number-in-source) or compare against a final reference value cannot distinguish between a simple typo and a deep semantic misunderstanding.
Methods
The authors introduce NumTrace, a benchmark designed to audit the source and financial semantics of every numerical claim an agent makes.
The core of NumTrace is a three-verdict auditing system:
- Value Check: Do the digits and arithmetic match the cited source?
- Binding Check: Does the number refer to the specific entity, concept, period, unit, and scope that the agent claims?
- Support Check: Does the cited source actually back both the value and the binding?
To evaluate agents, the researchers developed a standardized four-stage pipeline: Retrieve (using BM25), Extract (quoting lines), Calculate (applying formulas/scales), and Synthesize (writing the final report). This allows them to pinpoint exactly where in the reasoning chain an error first appears.
Datasets
NumTrace utilizes a frozen snapshot of 2,163 SEC EDGAR documents (10-K, 10-Q, Form 13F, and proxy statements) covering 30 large U.S. companies and 30 institutional managers.
The benchmark consists of 540 questions and 2,306 answer slots. Notably, it is the first benchmark to tackle:
- Form 13F: Complex reports on institutional share holdings.
- Item 402(i): Detailed tables regarding executive deferred compensation.
Reference answers were generated using 216 deterministic Python rules based on actual SEC instructions and accounting standards, ensuring a "ground truth" that accounts for nuances like unit changes over time or co-filing group memberships.
Key Insights and Ideas
- The RNWA Trap: The most common failure isn't "hallucinating" digits out of thin air; it's "hallucinating" the context. An agent might see the number "256" in a filing and report it as a group's total holding, when it actually only represents one small member of that group.
- Audit the Provenance, Not Just the Answer: By requiring a "provenance record" for every number, NumTrace forces agents to show their work. This makes the AI's reasoning "certifiable"—a human or another system can verify if the evidence cited actually supports the claim made.
- Complexity of Financial Semantics: Financial documents are "traps" for LLMs. For example, Form 13F changed its reporting units from thousands of dollars to single dollars in early 2023. Agents that don't account for this date-dependent rule will consistently fail the binding check on units, even if they copy the digits perfectly.
Key Results
The authors audited three state-of-the-art agents: Claude Opus 5.5, GPT-6 Astra, and DeepSeek-V4.1-Flash.
- RNWA Dominance: RNWA errors accounted for 67.5% to 81.8% of all failing claims across the three agents.
- High Fidelity, Low Accuracy: While agents passed the Value Check for over 91% of claims (meaning they are excellent at copying), their Binding Check pass rates were significantly lower, particularly on 13F filings (ranging from 65.4% to 82.7%).
- Group Totals are the Hardest: On questions involving co-filing groups (13F-GROUP), binding pass rates plummeted to between 41.9% and 48.5%. Agents frequently reported a single filer's position as the total for the entire group.
Limitations and Future Work
- Retrieval Bottlenecks: The authors noted that missing evidence in the retrieval stage accounted for a portion of the 13F errors, suggesting that better search mechanisms are needed alongside better reasoning.
- Scorer Sensitivity: While the automatic scorer is highly consistent, human expert review found that it still has a false-fail rate of about 20% for binding, often due to the nuances of natural language synthesis.
- Toward Runtime Certification: The authors suggest that the next step is developing "runtime certifiers"—systems that prevent an agent from releasing a numerical claim unless its provenance record can be independently verified against filing rules at the moment of generation.
Summary updated