M2P2: Multimodal Persuasion Prediction Using Adaptive Fusion

Chongyang Bai, Haipeng Chen, Srijan Kumar, Jure Leskovec, V. S. Subrahmanian

TL;DR

This paper introduces M2P2, a novel multimodal framework for predicting persuasion in debates. It tackles both predicting the debate's winner (DOP) and the subtle shift in audience votes for individual speakers (IPP), a problem previously unaddressed. M2P2 achieves state-of-the-art results by adaptively fusing information from audio, visual, and language modalities, effectively handling both aligned and conflicting signals.

What Problem Does This Paper Address?

Identifying persuasive speakers is crucial in various scenarios, from political campaigns to corporate advocacy. Debates are a common arena for such persuasion. While predicting the overall winner of a debate (Debate Outcome Prediction - DOP) has been studied, this paper highlights a more nuanced problem: predicting the Intensity of Persuasion (IPP). IPP focuses on how much a speaker influences individual audience members' opinions, measured by the change in their votes. This is critical because a speaker might not win the overall debate but could still be highly persuasive by swaying a significant portion of the audience away from their opponent. Existing methods for DOP often fail to fully leverage the rich, multimodal nature of debate data, particularly the semantic alignment and diverse information present across different modalities (audio, visual, text).

Methods

M2P2 employs an adaptive fusion framework to tackle the challenges of multimodal persuasion prediction. The core innovation lies in its ability to balance shared (aligned) and distinct (heterogeneous) information from different modalities.

  1. Input Embeddings: Audio, visual (facial features), and language (text) are processed to generate initial embeddings.
  2. Compact Latent Embeddings: A Transformer encoder is used for each modality to capture temporal dynamics and create compact latent representations.
  3. Adaptive Fusion: This is the heart of M2P2, comprising two key modules:
  4. Alignment Module: A shared MLP projects latent embeddings into a common space, and a loss function (cosine loss and domain adaptation loss) encourages alignment across modalities. This module captures shared information.
  5. Heterogeneity Module: This module learns to weight modalities based on their individual performance. It uses unimodal reference models to estimate "errors" and adaptively assigns weights to each modality's latent embedding before concatenation. This allows the model to down-weight noisy or less informative modalities.
  6. Final Prediction: The fused representation from the alignment and heterogeneity modules, along with debate meta-data (like initial vote counts), is fed into a final MLP to predict the outcome (DOP) or persuasion intensity (IPP).

Datasets

  • IQ2US Dataset: A US debate TV show dataset previously used for DOP. It provides debate outcomes but lacks intermediate vote data, making it suitable only for DOP.
  • QPS Dataset: A new dataset introduced by the authors, collected from the popular Chinese debate TV show "Qipashuo." This dataset is crucial as it provides real-time audience vote data before and after each speaker's turn, enabling the training and evaluation of the IPP problem. The authors describe a meticulous process for extracting transcripts from video subtitles, including image binarization, clustering, and OCR.

Key Insights and Ideas

  • The Importance of "Intensity": The paper convincingly argues that predicting the change in individual votes (IPP) is a distinct and valuable problem beyond just predicting the overall debate winner (DOP).
  • Adaptive Fusion is Key: Simply concatenating or averaging features from different modalities is insufficient. M2P2's adaptive fusion, which explicitly models both alignment and heterogeneity, is crucial for robust performance.
  • Handling Modality Noise: The heterogeneity module, by learning modality weights based on unimodal performance, effectively down-weights noisy or less informative modalities (e.g., visual noise, incorrect subtitles), as demonstrated in Figure 3.
  • Transformer's Temporal Power: The use of Transformer encoders proves effective in capturing long-range temporal dynamics within each modality, outperforming LSTM-based approaches.
  • Language Dominance: While all modalities contribute, language consistently emerges as the most important modality for persuasion prediction, though visual and audio cues also play significant roles.

Key Results

  • IPP Performance: M2P2 outperforms four recent baselines on the QPS dataset for IPP by at least 26.7% on average MSE, a statistically significant improvement.
  • DOP Performance: M2P2 improves DOP accuracy on the IQ2US dataset by 1.7%-2.5% compared to baselines, also statistically significant.
  • Ablation Studies: Removing the alignment module or the reference models in the heterogeneity module significantly degrades performance, highlighting their importance. Replacing the Transformer with an LSTM results in the worst performance, underscoring the Transformer's advantage.
  • Modality Importance: The language modality consistently shows the highest learned weight in the heterogeneity module, followed by visual and then audio.

Limitations and Future Work

  • Generalizability: The authors suggest that the adaptive fusion technique could be generalized to other multimodal sequence prediction tasks like video question answering and sentiment analysis.
  • Sample-Specific Weights: Future work could explore learning sample-specific modality weights using attention mechanisms for further performance gains.
  • Prior Knowledge: The current work uses publicly available TV show data. Future research could investigate the impact of prior audience knowledge by conducting controlled experiments.
  • Real-World Persuasion: The authors acknowledge that factors like logic and argument structure might be more critical in real-world persuasion scenarios than in TV debates. Future work could explore IRB-approved experiments with such data.

Summary updated

Abstract

Identifying persuasive speakers in an adversarial environment is a critical task. In a national election, politicians would like to have persuasive speakers campaign on their behalf. When a company faces adverse publicity, they would like to engage persuasive advocates for their position in the presence of adversaries who are critical of them. Debates represent a common platform for these forms of adversarial persuasion. This paper solves two problems: the Debate Outcome Prediction (DOP) problem predicts who wins a debate while the Intensity of Persuasion Prediction (IPP) problem predicts the change in the number of votes before and after a speaker speaks. Though DOP has been previously studied, we are the first to study IPP. Past studies on DOP fail to leverage two important aspects of multimodal data: 1) multiple modalities are often semantically aligned, and 2) different modalities may provide diverse information for prediction. Our M2P2 (Multimodal Persuasion Prediction) framework is the first to use multimodal (acoustic, visual, language) data to solve the IPP problem. To leverage the alignment of different modalities while maintaining the diversity of the cues they provide, M2P2 devises a novel adaptive fusion learning framework which fuses embeddings obtained from two modules -- an alignment module that extracts shared information between modalities and a heterogeneity module that learns the weights of different modalities with guidance from three separately trained unimodal reference models. We test M2P2 on the popular IQ2US dataset designed for DOP. We also introduce a new dataset called QPS (from Qipashuo, a popular Chinese debate TV show ) for IPP. M2P2 significantly outperforms 4 recent baselines on both datasets.
BibTeX
@article{bai2020m,
  author = {Chongyang Bai and Haipeng Chen and Srijan Kumar and Jure Leskovec and V. S. Subrahmanian},
  title = {{M2P2: Multimodal Persuasion Prediction Using Adaptive Fusion}},
  journal = {IEEE Trans. Multimedia},
  year = {2020},
  doi = {10.1109/tmm.2021.3134168},
}
Figure