Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt

Peter Anderson, Mano Vikash Janardhanan, Jason He, Wei Cheng, Charlie Flanagan

Abstract

General-purpose text embedding models often struggle with the specialized vocabulary and semantic nuances of the financial sector. This paper introduces BAM embeddings, a suite of models specifically finetuned on a massive dataset of 14.3 million financial query-passage pairs. BAM achieves a 62.8% Recall@1 on financial retrieval tasks, significantly outperforming general-purpose models like OpenAI’s text-embedding-3-large (39.2%). We detail the methodology of synthetic query generation, hard negative mining, and weight averaging that enables this performance leap.

Introduction

The financial domain is a "jungle" of arcane jargon and semantic collisions. Terms like "Greenback bear" (an investor betting against the USD) or "Chinese wall" (information barriers) have specific meanings that diverge from common internet usage. Furthermore, common words such as "Apple," "Short," or "Forward" refer to specific entities or instruments rather than their literal definitions. Existing NLP benchmarks often overlook these complexities, focusing instead on news or social media. For Retrieval-Augmented Generation (RAG) systems to be effective for financial analysts, embeddings must be adapted to this specialized vocabulary.

Methodology

The BAM embeddings are built upon the Multilingual-E5 model architecture (XLM-RoBERTa). The training process involved several key innovations:

  • Synthetic Query Generation: Using Mistral 7B, the authors generated queries for 15 million passages, seeded with human-written examples to maintain realism.
  • Hard Negative Mining: The model was trained to distinguish between correct passages and semantically similar but incorrect "hard negatives," providing a +5.3% boost in recall.
  • Model Soups (Weight Averaging): Weight averaging across multiple finetuned checkpoints and the base model was used to prevent overfitting and improve out-of-domain generalization.
  • Context Augmentation: Metadata such as tickers and company names were prepended to passages to provide necessary context for generic-sounding text.

Datasets

The primary training set consists of 14.3 million query-passage pairs (6 billion tokens) derived from a corpus of 2.8 million documents, including equity research, macro research, EDGAR filings (10-Ks/10-Qs), and earnings transcripts. Evaluation was conducted on a held-out test set of 447K pairs, as well as public benchmarks like FinanceBench and FiQA 2018.

Key Insights: The Advantage of Specificity

A notable finding is the "Long Query Advantage." While traditional lexical search (BM25) performance tends to degrade as queries become more detailed, BAM embeddings show improved performance with longer, more specific natural language questions. This shift encourages users to provide more context, leading to more accurate retrieval.

Recall@1 vs. Query Length for Lexical and Vector Search

Results

BAM-base (768-dim) achieved a Recall@1 of 62.8% on the financial test set. In comparison, OpenAI’s text-embedding-3-large (3072-dim) reached only 39.2%, and the older ada-002 scored 38.4%. In downstream RAG applications, replacing general-purpose embeddings with BAM resulted in an 8% increase in question-answering accuracy on FinanceBench.

Retrieval Performance Comparison: BAM vs. OpenAI and Open Source

Conclusion

BAM embeddings demonstrate that domain-specific finetuning at scale is essential for high-stakes financial applications. Future work will address the challenges of table extraction and temporal drift in financial language. While the training data remains proprietary, the models are available for public use on Hugging Face.

Summary updated

Abstract

Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. Yet, few text embeddings specialized for finance have been reported in the literature, perhaps in part due to a lack of public datasets and benchmarks. We present BAM embeddings, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents. Demonstrating the benefits of domain-specific training, BAM embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embedding from OpenAI. Further, BAM embeddings increase question answering accuracy by 8% on FinanceBench and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries. To support further research we describe our approach in detail, quantify the importance of hard negative mining and dataset scale, and publicly release our embeddings.
BibTeX
@inproceedings{anderson2024greenback,
  author = {Peter Anderson and Mano Vikash Janardhanan and Jason He and Wei Cheng and Charlie Flanagan},
  title = {{Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.26},
}
Figure