TL;DR
FinVerify is a novel verification framework that uses a combination of LLMs, deterministic rules, and SMT solvers to audit the numerical claims of financial AI agents. By categorizing errors into five specific financial protocols, it significantly boosts the precision of agent outputs, achieving up to 99.8% precision on SEC filing benchmarks.
What Problem Does This Paper Address?
Financial AI agents are increasingly used to parse SEC filings, calculate ratios, and support investment decisions. However, these agents often suffer from "Right Number, Wrong Account" (RNWA) errors—for example, pulling the correct revenue figure but for the wrong fiscal year, or using an incompatible unit of measurement.
Current verification methods fall short: general-purpose claim verifiers don't understand accounting identities, and "LLM-as-a-judge" approaches act as black boxes that can't provide a machine-checkable proof of their verdict. FinVerify addresses this by creating a structured, evidence-grounded framework that moves beyond simple text matching to verify the underlying financial logic and temporal context of a claim.
Methods
The authors propose a Proposal–Certification–Execution (PCE) architecture. FinVerify acts as the "Certification" layer through a multi-step process:
- Taxonomy of Errors: Based on an analysis of 190 agent failures, the authors identified five core error categories: Temporal Integrity, Source Fidelity, Comparability, Accounting Integrity, and Financial-Model Fidelity.
- Evidence-Bound Fitting: An LLM (Claude Haiku 4.5) extracts the agent's claims, the cited evidence, and the financial context (period, unit, entity) into a structured record.
- Deterministic Validation: Fixed rules check if the extracted quotes actually exist in the source text and if the metadata (like dates) align.
- Symbolic Reasoning (Z3): For complex accounting relationships (e.g., checking if a subtotal matches its components), FinVerify uses the Z3 SMT solver. It checks if the reported value is mathematically consistent with grounded inputs and authoritative formulas (like those found in XBRL linkbases).
- The Proof Gate: The system returns a verdict of Accept, Reject (if a violation is proven), or Abstain (if evidence is insufficient). A claim is only accepted if it passes every applicable protocol.
Datasets
The framework was evaluated on five benchmarks using fixed candidate sets:
- Agent Finance Reasoning: 357 multi-step trajectories from five different backbone models.
- XBRLFiling: A dataset focused on financial statement subtotals filed with the SEC.
- FinanceBench-67: QA based on annotated passages from SEC filings.
- FinQA & TAT-QA: Standard financial question-answering benchmarks consisting of text and tables.
Key Insights and Ideas
- Context is King: The study found that 76% of agent errors were related to "Source Fidelity"—meaning the value was either missing from the source or inconsistent with it. Simply re-running an agent's code won't catch these errors if the agent fed the code the wrong data to begin with.
- The Power of "Abstain": Unlike LLM judges that are often forced to guess, FinVerify’s ability to abstain when evidence is missing is a major strength. This conservative approach is vital for high-stakes financial environments where a "false accept" is much more dangerous than a "false reject."
- Symbolic vs. Probabilistic: By using Z3, the framework moves from probabilistic guessing to symbolic proof. If the solver finds the constraints "UNSAT" (unsatisfiable), it provides a definitive, explainable reason why the agent's claim is mathematically impossible given the data.
Key Results
FinVerify consistently outperformed both a "Direct" baseline and an LLM judge:
- Precision Gains: On the Agent Finance Reasoning benchmark, it improved accepted-claim precision from 56.4% (LLM judge) to 76.7%.
- Near-Perfect Accuracy: On XBRLFiling, it achieved 99.8% precision and 97.0% decision accuracy, compared to just 58.0% accuracy for the LLM judge.
- Conservative Reliability: While FinVerify abstains more often, it significantly reduces "false accepts." For instance, on FinanceBench-67, it reached 87.9% precision (vs. 69.7% for the judge).
- Benchmark Sensitivity: The improvements were most dramatic when authoritative evidence (like XBRL data) was available. On simpler text-and-table tasks (FinQA), the precision gap was smaller.
Limitations and Future Work
- In-Sample Taxonomy: The five error categories were derived from the same dataset used for part of the evaluation. Further research is needed to see if this taxonomy covers the full spectrum of errors in other financial domains.
- Dependency on Structured Data: FinVerify performs best when it can access structured filings (XBRL). Its performance is more limited when dealing with "free-text" reference answers that lack explicit accounting identities.
- Future Directions: The authors suggest moving toward "trace-level" certification, where the verifier doesn't just check the final answer but audits every step of the agent's reasoning process in real-time.
Summary updated