TL;DR
This paper tackles the challenge of hallucination in Large Language Models (LLMs) when answering causal questions from financial documents. They found that while prompt engineering and few-shot learning offer some improvement, fine-tuning LLMs is crucial for mitigating hallucinations and achieving superior performance in this complex domain.
What Problem Does This Paper Address?
Extracting causal relationships from financial documents is vital for informed decision-making. This involves understanding what factors influence financial performance, such as economic shifts or market trends. The FinCausal 2025 competition specifically focuses on generative models for causal question answering, moving beyond simple span extraction to require coherent, contextually relevant answers. A major hurdle in this task is "hallucination," where LLMs generate answers that are not grounded in the provided text, leading to inaccurate or fabricated information. This paper investigates how to effectively address this hallucination problem in the context of financial causal Q&A.
Methods
The authors explored two main approaches for causal Q&A:
- Encoder-based Token Classification: This method, similar to previous work, uses models like BERT to classify tokens within a passage as belonging to a cause or effect. They employed a standard tokenization and concatenation approach, followed by cross-entropy loss calculation, with a loss mask to focus on relevant tokens.
- Decoder-based Generative Models: This approach leverages the generative capabilities of LLMs. They experimented with prompt optimization, iteratively refining prompts to guide the models. They also utilized few-shot learning, providing the models with a small number of examples to learn from. Crucially, they then moved to fine-tuning LLMs (specifically Llama 3.1) on a multilingual dataset. This fine-tuning process was key to minimizing hallucinations and improving performance. They also used Low-Rank Adaptation (LoRA) to speed up the fine-tuning process.
Datasets
The study used the FinCausal 2025 dataset, which comprises:
- English text segments from UK financial reports (2017).
- Spanish text segments from a corpus of Spanish financial annual reports (2014-2018).
Each entry in the dataset includes an open-ended question, a context passage, and an extractive answer.
Key Insights and Ideas
- Fine-tuning is King for Hallucination Mitigation: While prompt engineering and few-shot learning can help, they are insufficient on their own to consistently prevent LLMs from hallucinating in complex causal Q&A tasks. Fine-tuning is the critical step for achieving robust and accurate answers.
- Multilingual Models Excel: BERT models pre-trained on multiple languages (English and Spanish) demonstrated better generalization and question-answering abilities across languages compared to monolingual models, even after fine-tuning on specific sub-task data.
- Generative Models Offer Deeper Understanding: Fine-tuned generative models, particularly Llama, can go beyond simple span extraction to generate responses that reflect a more comprehensive understanding of causal relationships, including implicit ones.
- Few-Shot Learning's Nuances: While few-shot learning improves performance, the number of shots that yields the best results isn't always consistent across different models.
- Implicit Causality Remains a Challenge: Both extractive and generative models struggled with implicit causal relationships where explicit linguistic markers were absent, and with nested causality where multiple potential causes were present.
Key Results
The paper reports significant success in the FinCausal 2025 competition:
- Their fine-tuned multilingual Llama 3.1 70B model achieved top results for Spanish and second-best for English.
- Specifically, for the English evaluation set (498 samples), their best model achieved a SAS score of 0.9732 and an Exact Match score of 0.8637 (using GPT-40 as a tiebreaker).
- For the Spanish evaluation set (500 samples), their best model achieved a SAS score of 0.9841 and an Exact Match score of 0.8703.
- Fine-tuned generative models significantly outperformed extractive models in identifying explicit causal links.
Limitations and Future Work
- Annotation Guidelines: The authors note uncertainty in their error analysis due to the absence of specific annotation guidelines for extracting causal answers. They suggest extending existing guidelines (Razouk et al., 2024a) as a future direction.
- Logit Manipulation: Exploring techniques like logit manipulation could further enhance performance by directly influencing token probabilities.
- Evaluation Metrics: Developing evaluation methods that better integrate causal modeling standards is needed, as current metrics might not fully capture the nuances of extracted causal information.
- Mixture of Experts: The success of using an additional model (GPT-40) as a tiebreaker suggests promising potential for future "mixture of expert" solutions tailored for extractive Q&A tasks.
Summary updated