DocFin: Multimodal Financial Prediction and Bias Mitigation using Semi-structured Documents

Puneet Mathur, Mihir Goyal, Ramit Sawhney, Ritik Mathur, Jochen L. Leidner, Franck Dernoncourt, Dinesh Manocha

Introduction

Predicting stock price movement and volatility is a cornerstone of financial analysis, yet it remains notoriously difficult due to the stochastic nature of markets. Current state-of-the-art models primarily rely on multimodal data from earnings calls, integrating text transcripts with audio features to capture executive sentiment. However, this approach suffers from two significant limitations: an information gap regarding hard financial data and inherent algorithmic bias stemming from gender-specific acoustic features.

The DocFin framework addresses these challenges by integrating semi-structured tabular data from official SEC filings (10-K and 10-Q reports) with traditional text and audio modalities. This integration not only provides a more grounded factual context for predictions but also offers a pathway to mitigate gender-based biases prevalent in audio-centric models.

Methods: The DocFin Framework

The core of the DocFin framework is a generalized method to fuse three distinct modalities: Text, Audio, and Tables.

Figure 1: The DocFin architecture illustrating the fusion of text, table, audio, and speaker embeddings through the DocEmbedding mechanism and a Transformer-based prediction head.

Table Linearization and DocEmbedding

To process semi-structured data, the researchers "linearize" table rows into natural language sentences and encode them using BERT. The technical centerpiece, DocEmbedding, utilizes a dot-product attention mechanism. This allows each spoken utterance in a transcript to "pull" relevant contextual information from the linearized financial tables, effectively grounding subjective executive claims in objective financial facts.

Multimodal Fusion and Bias Mitigation

The model calculates a weighted sum of text, audio, and table features. These are combined with speaker identities and positional embeddings before being processed by a Transformer to predict volatility (regression) and price movement (binary classification). Crucially, the researchers explored a "Text + Table" configuration as a bias-free alternative to "Text + Audio," hypothesizing that tables provide equivalent or superior predictive power without the demographic baggage of voice recordings.

Datasets

The study utilized two primary datasets augmented with SEC-EDGAR filings:

  1. MAEC: S&P 500 Earnings Calls.
  2. M3A: Merger & Acquisition conference calls.
  3. SEC-EDGAR: The authors scraped 10-K and 10-Q reports, specifically parsing HTML to retrieve tables with at least 10 rows to ensure high information density.

Key Results and Insights

The inclusion of tabular data through DocEmbedding yielded significant improvements across all metrics.

Figure 4: Performance comparison of DocFin (M3ANet + DocEmbedding) against state-of-the-art baselines for volatility and price prediction.
  • Performance Gains: Adding DocEmbedding to baseline models resulted in an 8-12% relative improvement in predicting stock volatility and price movement. As shown in the results, the DocFin-augmented model consistently outperformed previous state-of-the-art methods across 3, 7, and 15-day prediction windows.
  • Bias Reduction: One of the most striking findings was the reduction of gender-induced bias. Audio-based models often learn "imperfect generalizations" due to the under-representation of female executives. By substituting or supplementing audio with tabular data, the researchers reduced gender-based error disparities by over 30%.
  • Efficiency: Beyond accuracy and fairness, the "Text + Table" approach is highly efficient. Replacing audio clips with tabular data reduces storage requirements by 50% and processing time by over 90%, as it bypasses complex acoustic feature extraction and alignment.

Conclusion

DocFin demonstrates that semi-structured financial documents are a vital, yet underutilized, resource for market forecasting. By grounding multimodal models in tabular facts, researchers can achieve higher predictive accuracy while simultaneously building fairer, more efficient algorithmic systems. Future work will explore advanced document-NLP techniques, such as temporal dependency parsing, to better capture the evolution of financial data over time.

Summary updated

BibTeX
@inproceedings{mathur2022docfin,
  author = {Puneet Mathur and Mihir Goyal and Ramit Sawhney and Ritik Mathur and Jochen L. Leidner and Franck Dernoncourt and Dinesh Manocha},
  title = {{DocFin: Multimodal Financial Prediction and Bias Mitigation using Semi-structured Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.139},
}
Figure