Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt

Peter Anderson

TL;DR

This paper introduces BAM embeddings, a new set of text embeddings specifically trained for the financial domain. By fine-tuning on a massive dataset of financial queries and passages, BAM embeddings significantly outperform general-purpose embeddings in financial document retrieval and question answering, demonstrating the critical need for domain-specific adaptations in NLP.

What Problem Does This Paper Address?

The financial industry is awash in specialized jargon, acronyms, and technical terms that general-purpose text embeddings struggle to interpret. This makes it difficult for information retrieval and RAG systems to effectively extract insights from vast amounts of financial documents. Existing benchmarks and datasets are also lacking for evaluating finance-specific text embeddings, hindering progress in this area. The authors aim to bridge this gap by creating and evaluating domain-specific embeddings for finance.

Methods

The core of the paper is the development of BAM embeddings, which are built upon the Multilingual-E5 model. The key innovation lies in the extensive fine-tuning process using a carefully curated dataset of 14.3 million query-passage pairs. This dataset was constructed using a novel synthetic query generation strategy:

  1. Passage Sampling: Text passages were extracted from 2.8 million financial documents, parsed, and split into manageable chunks. Document context (company name, ticker, event) was prepended to each passage.
  2. Synthetic Query Generation: An LLM (Mistral 7B Instruct) was prompted to generate queries for each passage. This process was seeded with human-written queries to guide the LLM towards generating relevant and diverse queries.
  3. Hard Negative Mining: To improve embedding quality, the authors incorporated "hard negatives" – passages that are similar but not correct answers – into the training process. This technique, along with scaling the dataset, was found to be crucial for performance gains.
  4. Weight Averaging: For deployment, the authors averaged the parameters of multiple fine-tuned checkpoints with the baseline model to enhance robustness and accuracy.

Datasets

  • Training Data: A custom dataset of 14.3 million query-passage pairs, constructed from 2.8 million public and proprietary financial documents.
  • Evaluation Data:
    • A held-out test set of 447,000 query-passage pairs for passage retrieval evaluation.
  • FinanceBench (Islam et al., 2023) for question answering evaluation in a RAG setting.

Key Insights and Ideas

  • Domain Specificity is Crucial: The significant performance gap between BAM embeddings and general-purpose models highlights that financial language requires specialized understanding.
  • Synthetic Query Generation is Scalable and Effective: The authors demonstrate that LLM-driven synthetic query generation can create large, high-quality datasets for training specialized embeddings, overcoming the limitations of relying solely on scarce human-generated data.
  • Hard Negative Mining is Key: Incorporating challenging negative examples during training is vital for pushing the boundaries of retrieval performance.
  • Dataset Scale Matters: Increasing the size and diversity of the training data directly translates to improved embedding quality.
  • Vector Search Outperforms Lexical Search on Complex Queries: As queries become longer and more detailed, BAM embeddings show improved performance, while traditional lexical search (BM25) degrades.

Key Results

  • Passage Retrieval: BAM embeddings achieve 62.8% Recall@1 on a held-out test set, significantly outperforming OpenAI's text-embedding-3-large model (39.2%) and the base Multilingual-E5 model (34.3%).
  • Question Answering (FinanceBench): Replacing OpenAI's ada-002 embeddings with BAM embeddings in a RAG system improved question answering correctness by 8%, from 47% to 55%.
  • Ablation Studies:
    • Hard negative mining improved Recall@1 by 5.3%.
  • Scaling the dataset (using the full 14.3M pairs vs. 37% of it) improved Recall@1 by 4.5%.
  • Qualitative Analysis: Finetuned embeddings are more sensitive to company tickers and financial metrics, leading to better performance on detailed, forward-looking, and company-specific queries.

Limitations and Future Work

  • Dataset Release: The authors cannot release their curated dataset due to proprietary data.
  • Table Data: The current work focuses on text passage retrieval, and information extraction from tables within financial documents is left for future work.
  • LLM Errors: In the FinanceBench evaluation, remaining errors were often attributable to the LLM's inability to extract numbers from tables or perform complex calculations.
  • Future Work: The authors suggest further research into improving query generation strategies and exploring the application of BAM embeddings in other financial NLP tasks. They also highlight the potential for using their embeddings to improve query completion and autocomplete features in user-facing applications.

Summary updated

Abstract

Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. Yet, few text embeddings specialized for finance have been reported in the literature, perhaps in part due to a lack of public datasets and benchmarks. We present BAM embeddings, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents. Demonstrating the benefits of domain-specific training, BAM embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embedding from OpenAI. Further, BAM embeddings increase question answering accuracy by 8% on FinanceBench and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries. To support further research we describe our approach in detail, quantify the importance of hard negative mining and dataset scale, and publicly release our embeddings.
BibTeX
@inproceedings{anderson2024greenback,
  author = {Peter Anderson},
  title = {{Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.26},
}
Figure