Multimodal Multi-Speaker Merger & Acquisition Financial Modeling: A New Task, Dataset, and Neural Baselines

Ramit Sawhney, Mihir Goyal, Prakhar Goel, Puneet Mathur, Rajiv Ratn Shah

In the financial sector, Merger & Acquisition (M&A) announcements represent high-stakes events where investors look beyond press releases to the nuances of conference calls. While Natural Language Processing (NLP) has traditionally analyzed these transcripts, it often overlooks the critical dimensions of vocal cues and speaker identity. New research introduces the $M^3A$ dataset and $M^3ANet$, a "Speaker Transformer" designed to bridge this gap by integrating text, audio, and speaker context to forecast market reactions.

The $M^3A$ Dataset

The authors curated the Multimodal Multi-Speaker Merger & Acquisition ($M^3A$) dataset, the first of its kind to align audio with text for these specific financial events. Sourced from Bloomberg Terminal and Yahoo Finance, the dataset includes 816 M&A calls spanning 2016 to 2020, totaling over 545 hours of audio.

Yearly frequency and characteristics of the M3A dataset, showing trends in call volume, utterances, speakers, and audio length.

Unlike standard earnings call datasets, $M^3A$ focuses on the high-volatility environment of M&As. The data is granularly segmented at the utterance level and forced-aligned, ensuring precise synchronization between text snippets and audio timestamps. As shown in the dataset statistics, the volume of acquisition calls significantly outweighs mergers, and the average characteristics (utterances, speakers, length) vary across the five-year period.

Methods: The $M^3ANet$ Architecture

The proposed $M^3ANet$ is a neural architecture specifically engineered to process multi-speaker, multimodal inputs. The pipeline moves financial modeling from "what was said" to "how it was said and by whom."

High-level architecture of M3ANet, illustrating the fusion of speaker annotations, text, and audio embeddings into a Speaker Enhanced Transformer for financial prediction.

The model's workflow consists of four primary stages:

  1. Multimodal Encoding: Text is processed via BERT for semantic meaning, while audio features (pitch, jitter, loudness) are extracted using OpenSMILE.
  2. Attention-Fusion: Rather than simple concatenation, an attention mechanism dynamically weighs the importance of text versus audio based on input quality.
  3. Speaker Information Infusion: The model's "secret sauce" involves assigning unique IDs to roles (e.g., CEO, Analyst, Operator). This allows the model to weigh a CEO's statement differently than an analyst's question.
  4. Sentence-Level Transformer: These fused features, augmented with positional and speaker embeddings, pass through a Transformer block to model the temporal flow of the entire call.

Key Insights and Results

The $M^3ANet$ was evaluated on two tasks: Stock Volatility (regression) and Price Movement (binary classification).

  • The "Who" Matters: Incorporating speaker context consistently improved performance. The model successfully differentiated between management (the decision-makers) and analysts (the risk-gaugers).
  • Vocal Indicators: Qualitative analysis revealed that sudden peaks in a CEO’s pitch during Q&A sessions often preceded stock price drops, suggesting that vocal indicators of low confidence serve as leading indicators of market skepticism.
  • Predictive Performance: $M^3ANet$ achieved the lowest Mean Squared Error (MSE) for volatility across 3, 7, and 15-day windows. For price movement, it reached a best-in-class F1 score of 0.59, outperforming unimodal BERT or GloVe baselines.
  • Volatility Drift: The model found medium-term volatility (15 days) easier to predict than short-term (3 days), indicating that immediate post-call market noise eventually settles into more predictable trends.

Limitations and Future Work

While $M^3ANet$ establishes a strong baseline, the authors note several challenges. The dataset is currently limited to U.S.-based, English-speaking executives, which may introduce demographic and gender biases. Future iterations could incorporate graph neural networks to model complex inter-company relationships, further refining accuracy in this inherently difficult forecasting task.

Summary updated

BibTeX
@inproceedings{sawhney2021multimodal,
  author = {Ramit Sawhney and Mihir Goyal and Prakhar Goel and Puneet Mathur and Rajiv Ratn Shah},
  title = {{Multimodal Multi-Speaker Merger \& Acquisition Financial Modeling: A New Task, Dataset, and Neural Baselines}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.526},
}
Figure