FIRE: A Dataset for Financial Relation Extraction

Hassan Hamad, Abhinav Kumar Thakur, Nijil Kolleri, Sujith Pulikodan, Keith M. ChuggNAACL 2024 · DOI

TL;DR

This paper introduces FIRE, a novel, hand-annotated dataset for financial relation extraction, designed to address the lack of comprehensive resources in this domain. By including 13 entity types and 18 relation types, FIRE aims to facilitate the development and evaluation of advanced NLP models for understanding complex financial information.

What Problem Does This Paper Address?

Extracting structured information, specifically named entities and their relationships, from unstructured financial text is a critical but challenging task. Existing datasets for relation extraction (RE) often lack domain specificity or suffer from noisy annotations due to methods like distant supervision. This gap hinders the development of robust NLP models capable of accurately interpreting financial documents like SEC filings and news articles, which are crucial for financial analysis and decision-making. The authors highlight that even specialized financial NLP models like BloombergGPT lack dedicated evaluation benchmarks.

Methods

The core contribution is the creation of the FIRE dataset, which is meticulously hand-annotated by a single annotator to ensure consistency and minimize noise. The dataset was built using a semi-automated process: an initial "seed" dataset was hand-labeled, then used to train a joint NER and RE model. This model then suggested potential sentences for annotation, with a focus on low-confidence predictions to mitigate bias. The authors also developed an open-source, web-based labeling tool to aid in efficient and precise annotation. A unique feature of FIRE is the inclusion of "labeling time" for each instance, which the authors propose as a proxy for example difficulty and explore its use in curriculum learning strategies.

Datasets

The primary dataset introduced is FIRE (FInancial Relation Extraction), comprising 3,025 instances. It features 13 named entity types and 18 relation types, sourced from financial reports (SEC filings) and financial news articles. The paper also references existing datasets for comparison, including:

  • FinRED: An RE dataset for the financial domain, but labeled using distant supervision.
  • KPI-EDGAR: A joint NER and RE dataset focused on Key Performance Indicators.
  • General-purpose RE datasets like TACRED and DocRED.

Key Insights and Ideas

  • The value of hand-annotation: The paper strongly advocates for hand-annotated datasets in specialized domains like finance, arguing that it leads to higher quality and more reliable data compared to distant supervision.
  • Labeling time as a difficulty proxy: The inclusion of labeling time per instance is a novel approach to capture annotation difficulty, which can be leveraged for more effective model training through curriculum learning.
  • Curriculum learning benefits: Experiments show that curriculum learning, guided by labeling time, can enhance the performance of RE models, particularly for more challenging relation types.
  • RE-specialized models still dominate: Despite advancements in general-purpose LLMs, specialized RE models continue to outperform them on complex tasks like financial relation extraction.

Key Results

  • The FIRE dataset contains 3,025 instances, 13 entity types, and 18 relation types.
  • RE-specialized models (SpERT, PL-Marker) achieved micro F1 scores around 67-68% for relation extraction on the FIRE dataset.
  • Curriculum learning improved the performance of RE-specialized models by a small but consistent margin (e.g., SpERT's relation F1 increased from 67.41% to 68.11%).
  • General-purpose LLMs like GPT-3.5 and Llama 2-7b significantly underperformed RE-specialized models, especially in few-shot settings, though fine-tuning improved their performance.
  • Inter-annotator agreement studies revealed the technical complexity of the task, reinforcing the decision to use a single annotator for the final dataset.

Limitations and Future Work

  • Domain Specificity: FIRE is exclusively focused on the financial sector, limiting its applicability to other domains.
  • Language Restriction: The dataset is only in English, restricting its use in multilingual or cross-lingual studies.
  • Single Annotator Bias: The dataset was annotated by a single human who is not a domain expert or linguist, potentially introducing subjectivity and biases.
  • Entity/Relation Coverage: Due to the complexity of financial language, some entities and relations might not be captured by the current categories.
  • Future Work: The authors suggest expanding the dataset's domain knowledge and linguistic diversity. Further research could explore more advanced curriculum learning techniques and investigate how to better adapt general-purpose LLMs for financial relation extraction.

Summary updated

BibTeX
@inproceedings{hamad2024fire,
  author = {Hassan Hamad and Abhinav Kumar Thakur and Nijil Kolleri and Sujith Pulikodan and Keith M. Chugg},
  title = {{FIRE: A Dataset for Financial Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.230},
}
Figure