When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain

Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, Diyi YangEMNLP 2022 · DOI

Introduction

In the rapidly evolving field of Natural Language Processing (NLP), general-purpose models like BERT have set high standards for text understanding. However, the financial domain presents unique challenges. A standard model might interpret the phrase "crude oil prices are rising" as a neutral or positive statement about growth, whereas a financial expert recognizes it as a potential signal of increased operational costs or inflation. Finance is a specialized dialect where sentiment and intent are often embedded in technical jargon rather than emotive language.

To address this, researchers from Georgia Tech, JPMorgan AI Research, and Stanford University have introduced FLANG and FLUE. This contribution provides both a high-performance financial language model and a comprehensive benchmark suite designed to standardize evaluation in financial NLP.

The FLANG Framework

The researchers developed FLANG (Financial LANGuage model) using two distinct architectures: FLANG-BERT and FLANG-ELECTRA. Unlike previous attempts that simply continued training general models on financial text, FLANG introduces domain-specific learning objectives to capture the nuances of financial terminology.

Overview of the FLANG pretraining and FLUE finetuning pipeline.

Key Methodologies

  • Preferential Financial Masking: Rather than masking tokens at random, the model targets financial keywords and phrases from curated sources like Investopedia. The authors found that a ratio of 30% financial terms to 70% random tokens optimized performance.
  • Phrase-Level Learning: Financial concepts are often multi-token (e.g., "net cash flow"). FLANG masks these entire spans as a single unit, forcing the model to learn the underlying concept rather than predicting individual words.
  • Span Boundary Objective (SBO): By predicting masked spans using only the representations of adjacent words, the model develops a more robust understanding of context.
  • Supervised Contrastive Learning (SCL): During the fine-tuning phase, this loss function pulls similar examples closer in vector space, significantly improving classification accuracy.

The FLUE Benchmark

To measure progress effectively, the authors proposed FLUE (Financial Language Understanding Evaluation), a centralized benchmark analogous to GLUE but tailored for finance. FLUE encompasses five critical tasks:

  1. Financial Sentiment Analysis: Utilizing the FPB and FiQA SA datasets.
  2. News Headline Classification: Categorizing headlines based on market impact.
  3. Named Entity Recognition (NER): Identifying organizations, people, and financial instruments.
  4. Structure Boundary Detection: Parsing complex, messy financial filings to find sentence and table boundaries.
  5. Question Answering: Addressing opinion-based financial queries.

Experimental Results

FLANG-ELECTRA emerged as the state-of-the-art (SOTA) model across nearly all benchmarks. The improvements were particularly notable in sentiment-heavy tasks where domain expertise is paramount.

Performance comparison of FLANG models against BERT and FinBERT baselines across FLUE tasks.

As shown in the results table, FLANG-ELECTRA achieved an accuracy of 91.9% on the Financial PhraseBank (FPB), a significant jump from the 87.2% achieved by the previous FinBERT baseline. Furthermore, in the Question Answering task (FiQA QA), FLANG-ELECTRA reached an nDCG of 0.55, outperforming both BERT-base and FinBERT by a wide margin. These results suggest that phrase-level learning and preferential masking allow the model to "speak the language" of finance more fluently than its predecessors.

Conclusion and Future Directions

The introduction of FLANG and FLUE sets a new standard for financial NLP. While the current models show remarkable gains, the authors note several areas for future exploration. Current benchmarks focus on English and encoder-only architectures; expanding to multilingual datasets and generative (decoder-only) models like GPT could further enhance the utility of these tools for global markets. Additionally, incorporating informal data from social media platforms could help capture retail investor sentiment more accurately.

Summary updated

BibTeX
@inproceedings{shah2022when,
  author = {Raj Sanjay Shah and Kunal Chawla and Dheeraj Eidnani and Agam Shah and Wendi Du and Sudheer Chava and Natraj Raman and Charese Smiley and Jiaao Chen and Diyi Yang},
  title = {{When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.148},
}
Figure