Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models

Jaeyoung Choe, Keonwoong Noh, Nayeon Kim, Seyun Ahn, Woohwan JungEMNLP 2023 · DOI

Abstract

Specialized financial language models often underperform general-purpose models due to narrow training corpora. This paper introduces FiLM (Financial Language Model), a model pretrained on a highly diverse financial corpus encompassing SEC filings, earnings calls, news, and academic papers. FiLM achieves state-of-the-art performance across various financial benchmarks while utilizing significantly less data and energy than previous specialized models, demonstrating that corpus diversity is more critical than raw volume for domain-specific pretraining.

Introduction

In Natural Language Processing (NLP), domain-specific models typically outperform general-domain counterparts. However, the financial sector has remained an outlier; many "FinBERTs" struggle to surpass general models like RoBERTa. The authors identify a "diversity gap" as the primary cause. Most existing financial models are trained on limited sources, such as news or SEC filings alone. Given the linguistic variance between narrative news and dense financial reports, these models fail to generalize across the broader financial landscape.

Methodology

The authors developed FiLM by prioritizing data variety over volume. Starting with a RoBERTa-base architecture, they performed additional pretraining on a categorized financial corpus.

  1. Corpus Categorization: The financial domain was divided into five groups: News, SEC filings, Earnings Calls, Academic Papers, and Miscellaneous.
  2. Preprocessing: Aggressive deduplication removed 900 million redundant tokens, reducing the dataset from 3.3B to 2.4B tokens, which improved both accuracy and efficiency.
  3. Training: FiLM was trained for a single epoch, as performance was found to saturate quickly, leading to an 82% reduction in energy consumption compared to previous models.

Data and Evaluation

The training set comprises 2.4 billion tokens from 10 diverse sources.

Table 1: Overview of the diverse financial datasets used for pretraining FiLM.

Evaluation was conducted across six benchmarks:

  • FPB: Sentiment classification of news.
  • NER: Identifying entities in SEC contracts.
  • Headline: Classifying the impact of gold-related news.
  • FiNER: Identifying numeric entities in financial reports.
  • FinQA: Question answering requiring numerical reasoning.
  • FOMC: Classifying the stance of Federal Reserve communications.

Key Insights: Diversity Over Volume

A central finding of the research is that quality and variety outperform brute-force quantity. The researchers demonstrated that a model trained on 2.1B tokens from four different document groups outperformed a model trained on 3.1B tokens from a single group (SEC filings).

Figure 1: Average F1 score across tasks as the number of corpus groups increases, showing the benefit of diversity.

Furthermore, FiLM's strong performance on the FOMC task—despite not being explicitly trained on Fed transcripts—suggests that a diverse pretraining diet provides the "common sense" necessary for niche financial tasks.

Results

FiLM consistently outperformed existing financial PLMs and general-domain baselines:

  • Sentiment (FPB): 86.25% accuracy (vs. RoBERTa's 85.30%).
  • Numeric Reasoning (FinQA): 58.85% program accuracy, a significant improvement over the 30-40% range of previous models.
  • Macroeconomics (FOMC): F1 score of 69.60, surpassing SEC-BERT (65.07).

Conclusion and Future Work

While FiLM sets a new standard for financial encoders, numerical reasoning remains a challenge. As an encoder-only model, FiLM is optimized for classification and understanding rather than generative tasks. Future research should explore specialized tokenization for numerical data within the financial domain.

The authors have released FiLM and the associated code on GitHub and Hugging Face.

Summary updated

BibTeX
@inproceedings{choe2023exploring,
  author = {Jaeyoung Choe and Keonwoong Noh and Nayeon Kim and Seyun Ahn and Woohwan Jung},
  title = {{Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.138},
}
Figure