FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents

Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming Zhang, Xiangru Tang, Chen Zhao, Arman Cohan

TL;DR

This paper introduces FinDVer, a new benchmark for evaluating Large Language Models (LLMs) on their ability to verify claims within long and complex financial documents. Even state-of-the-art LLMs like GPT-4o struggle to match human expert performance, highlighting the challenges of this domain and the need for more robust evaluation.

What Problem Does This Paper Address?

In today's information-saturated world, verifying claims about companies' financial performance is crucial to prevent misinformation. This often requires sifting through lengthy and intricate financial documents like earnings reports and regulatory filings. While LLMs have shown promise in various tasks, applying them to this "document-grounded claim verification" in finance is challenging due to:

  1. Document Complexity: Financial documents are long, dense, and contain a mix of text and tables, requiring sophisticated comprehension skills and domain-specific knowledge.
  2. Reasoning Requirements: Verifying claims often involves not just extracting information but also performing numerical reasoning, integrating external knowledge, and providing clear, explainable rationales. Existing benchmarks often focus on simple entailment classification and don't adequately capture these complex reasoning processes.

Methods

The paper introduces FinDVer, a comprehensive benchmark designed to address these challenges. Its construction involved several key steps:

  • Expert Annotation: Financial experts were employed to ensure high data quality and relevance to real-world scenarios.
  • Hybrid-Content Documents: The benchmark utilizes long financial documents that combine both textual and tabular data.
  • Reasoning-Process Explanation: A core feature is the requirement for LLMs to generate detailed explanations of their reasoning process, not just an entailment label. This allows for a deeper understanding of how a model arrives at its conclusion.
  • Three Subsets: FinDVer is divided into three subsets, each targeting a specific reasoning challenge:
  • FDV-IE (Information Extraction): Focuses on extracting relevant information from text and tables.
  • FDV-MATH (Numerical Reasoning): Requires performing calculations or statistical analysis on extracted data.
  • FDV-KNOW (Knowledge-Intensive Reasoning): Involves integrating external domain-specific knowledge or regulations.
  • Two Evaluation Settings: LLMs are evaluated in two realistic settings:
  • Long-Context: The entire financial document is fed into the LLM.
  • Retrieval Augmented Generation (RAG): Relevant document snippets are retrieved and then fed to the LLM.
  • Chain-of-Thought (CoT) Prompting: The experiments primarily use CoT prompting to encourage LLMs to generate step-by-step reasoning before providing a final answer.

Datasets

The primary contribution is the FinDVer benchmark, which consists of 2,400 expert-annotated examples. These examples are derived from quarterly (Form 10-Q) and annual reports (Form 10-K) of companies, collected from the U.S. Securities and Exchange Commission. The benchmark is designed to be challenging, with long documents and a focus on explainability.

Key Insights and Ideas

  • LLMs Still Lag Human Experts: Even the most advanced LLMs, like GPT-4o, significantly underperform human experts in claim verification on financial documents, highlighting the complexity of the task.
  • Explainability is Crucial: Simply providing an entailment label is insufficient. The ability to generate coherent and accurate reasoning explanations is key for practical application and trust.
  • Domain-Specific Challenges: Financial documents present unique hurdles related to numerical reasoning, specialized terminology, and the integration of external financial regulations.
  • Long-Context vs. RAG: Some LLMs perform better in a pure long-context setting, suggesting architectural advancements are enabling better handling of extended inputs.
  • CoT Prompting Matters: Using Chain-of-Thought reasoning significantly improves LLM performance compared to direct output, underscoring the importance of structured reasoning.

Key Results

  • Human Performance: Human experts achieve an accuracy of 93.3% on the FinDVer benchmark, while the best LLM, GPT-4o, reaches only 76.2% in the RAG setting.
  • LLM Performance: Across various LLMs tested, performance varies, but none approach human-level accuracy. Open-source models like Llama-3.1-70B show promising results, even surpassing GPT-40 on the FDV-KNOW subset.
  • RAG vs. Long-Context: Figure 3 suggests that for some frontier LLMs, the long-context setting can yield better performance than RAG, indicating progress in handling long inputs.
  • Retrieval Quality: Table 4 shows that higher retrieval recall generally improves LLM performance in RAG settings, though the choice of retriever (e.g., BM25 vs. Contriever) also impacts results.
  • CoT Impact: Table 5 demonstrates that performance degrades when CoT prompting is removed, emphasizing its importance for this task.
  • Error Analysis: GPT-4o's errors fall into four main categories: extraction errors, numerical reasoning errors, domain knowledge deficiencies, and computation errors.

Limitations and Future Work

  • Limited Scope of LLMs: The evaluation did not include recently released finance-specific LLMs due to compatibility issues with the inference framework.
  • No Large-Scale Fine-tuning: The study did not perform large-scale fine-tuning of LLMs on finance-domain data, which could potentially improve performance, especially on FDV-KNOW.
  • Human Error Analysis: The error analysis was conducted on a small scale by human evaluators. The authors suggest future work could explore automated evaluation systems for detecting reasoning errors.
  • Future Directions: The paper highlights the need for further research into LLM capabilities for handling complex, expert-domain documents and suggests that domain-specific fine-tuning could be a promising avenue. The development of LLM-based automated evaluation systems is also recommended.

Summary updated

Abstract

We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents.
BibTeX
@inproceedings{zhao2024findver,
  author = {Yilun Zhao and Yitao Long and Yuru Jiang and Chengye Wang and Weiyuan Chen and Hongjun Liu and Yiming Zhang and Xiangru Tang and Chen Zhao and Arman Cohan},
  title = {{FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=E72Gh1TwMm},
}
Figure