WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, Diyi YangarXiv 2022 · DOI · arXiv

In the world of finance, language is a different beast. A phrase like "crude oil prices are going up" might sound like a neutral observation to a general-purpose AI, but to a financial analyst, it’s a signal of potential market stress. Standard language models often miss these nuances because they are trained on Wikipedia or books, where "up" is usually good and "down" is usually bad.

A team from Georgia Tech, JPMorgan, and Stanford recently released a paper that addresses this gap head-on. They introduced FLANG, a domain-specific pre-training methodology, and FLUE, a comprehensive benchmark suite designed to push the boundaries of Financial Natural Language Processing (FinNLP).

Methods: The FLANG Architecture

The authors introduced two primary models: FLANG-BERT and FLANG-ELECTRA. Their "secret sauce" lies in how the models are taught to understand text during pre-training. Unlike standard models that mask tokens randomly, FLANG uses a domain-specific approach to capture financial context.

Overview of the FLANG pre-training and FLUE fine-tuning pipeline.

Key innovations include:

  • Preferential Masking: Using a dictionary of over 8,200 financial terms, these keywords are given a 30% higher probability of being masked, forcing the model to master financial jargon.
  • Phrase-Level Masking: Multi-token financial phrases (e.g., "margin call") are masked as single units.
  • Span Boundary Objective (SBO): In the ELECTRA-based model, the system predicts masked spans using only surrounding context words, strengthening its grasp of term relationships.
  • Supervised Contrastive Learning (SCL): During fine-tuning, SCL helps the model cluster similar examples, improving its ability to distinguish subtle classes like "slightly bullish" versus "neutral."

Datasets and the FLUE Benchmark

The researchers curated a massive pre-training corpus including SEC filings (10-K/10-Q), earnings conference call transcripts, analyst reports, and news from Reuters and Bloomberg. To evaluate these models, they established the Financial Language Understanding Evaluation (FLUE) benchmark, which covers five distinct tasks:

  1. Sentiment Classification: Financial PhraseBank (FPB) and FiQA Sentiment.
  2. News Headline Classification: Categorizing headlines into 9 binary labels.
  3. Named Entity Recognition (NER): Identifying organizations and people.
  4. Structure Boundary Detection (FinSBD3): Detecting sentence and list boundaries in complex financial PDFs.
  5. Question Answering (FiQA QA): Retrieving relevant answers to financial queries.

Key Results

FLANG-ELECTRA emerged as the new heavyweight champion of FinNLP, consistently outperforming previous state-of-the-art models like FinBERT across the entire FLUE suite.

Performance comparison of FLANG models against BERT and FinBERT across the FLUE benchmark tasks.

Notable highlights include:

  • Sentiment Analysis (FPB): Achieved 91.9% accuracy, significantly outperforming FinBERT (87.2%).
  • Question Answering: On the FiQA QA task, FLANG-ELECTRA reached an nDCG of 0.55, a massive jump over FinBERT’s 0.42.
  • News Headlines: Achieved a Mean F1 of 0.98, setting a new state-of-the-art.
  • Perplexity: FLANG-BERT achieved a lower perplexity (19.53) than both standard BERT (23.66) and FinBERT (21.11), indicating it is much more "at home" with financial text.

Conclusion and Future Work

The paper proves that how you mask is as important as what you train on. By prioritizing financial terms and phrases, the FLANG models develop a sharper understanding of the domain. While a major step forward, the authors noted that future work should expand into generative tasks (like abstractive summarization), informal social media language (FinTwit), and multi-lingual support for global markets.

Summary updated

Abstract

Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.
BibTeX
@misc{shah2022when,
  author = {Raj Sanjay Shah and Kunal Chawla and Dheeraj Eidnani and Agam Shah and Wendi Du and Sudheer Chava and Natraj Raman and Charese Smiley and Jiaao Chen and Diyi Yang},
  title = {{WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain}},
  year = {2022},
  eprint = {2211.00083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.00083},
}
Figure