Strong and Simple Baselines for Multimodal Utterance Embeddings

Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency

Abstract

This paper challenges the prevailing trend of utilizing increasingly complex neural architectures for multimodal tasks. We introduce two simple, linear-based baselines for learning multimodal utterance embeddings: Multimodal Baseline 1 (MMB1) and MMB2. These models leverage weighted averages of features and closed-form solutions, achieving competitive performance with state-of-the-art deep learning models. Notably, our approach is over 10 times faster during inference and demonstrates superior robustness in low-data scenarios, suggesting that simple linear models can provide a powerful foundation for multimodal learning.

Introduction

Human communication is inherently multimodal, integrating text, visual cues (facial expressions, gestures), and acoustic signals (tone of voice). To enable machines to understand utterances, researchers strive to learn joint representations, or embeddings, that capture these diverse signals. Recent trends in the field have favored complex "black box" models, such as LSTMs, GRUs, and attention mechanisms. While effective, these models are computationally intensive, data-hungry, and often lack transparency. This research investigates whether such complexity is necessary or if simple linear models can serve as strong baselines.

Methodology

We propose two primary models: Multimodal Baseline 1 (MMB1) and MMB2.

MMB1, the Factorized Unimodal Model, assumes that the multimodal signal can be decomposed into independent unimodal factors conditioned on a joint embedding. Language is modeled using a log-linear word production model, while visual and acoustic signals are modeled as being generated from Gaussian distributions. A key mathematical contribution is the proof that the optimal joint embedding can be solved in closed form, resulting in a weighted average of features where weights are determined by word frequency and signal variance.

Overview of the multimodal utterance embedding process, illustrating the integration of language, visual, and acoustic modalities into a joint embedding.

MMB2 extends this by incorporating "early fusion" factors to capture interactions between modalities (e.g., bimodal and trimodal interactions), maintaining the efficient linear framework. Optimization is performed via a coordinate-ascent style algorithm, alternating between the closed-form embedding solution and gradient-based updates for linear transformation parameters.

Experimental Evaluation

The models were evaluated on two benchmark datasets: CMU-MOSI for multimodal sentiment analysis and POM (Persuasive Opinion Multimedia) for speaker trait recognition. These datasets provide a rigorous testing ground for sentiment and personality trait detection from video clips.

Results and Discussion

Our simple baselines demonstrated several key advantages over complex neural counterparts:

  • Competitive Performance: MMB2 matched or exceeded the accuracy of complex models like the Memory Fusion Network (MFN) and Tensor Fusion Network (TFN).
  • Efficiency: MMB1 and MMB2 achieved massive speedups, ranging from 10x to 100x in inferences per second (IPS). For instance, MMB1 processed 421,000 IPS compared to MFN's 4,760.
  • Robustness to Data Scarcity: In semi-supervised settings with only 40-60% of labels, our low-parameter baselines significantly outperformed deep autoencoders, which are prone to overfitting.
  • Distributional Assumptions: Empirical evidence supports the use of Gaussian likelihoods for non-verbal signals. Histograms of pre-extracted visual and acoustic features show they closely follow Gaussian distributions over time.
Empirical histograms of visual (green) and acoustic (blue) features, demonstrating their alignment with Gaussian distributions.

Conclusion

This work demonstrates that simple linear models, when properly formulated, can serve as exceptionally strong baselines for multimodal utterance embeddings. By prioritizing efficiency and transparency, these models offer a viable alternative to complex neural networks, particularly in resource-constrained or low-data environments. We suggest that these baselines should be the standard starting point for future multimodal research to ensure that added complexity provides genuine incremental value.

Summary updated

Abstract

Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.
BibTeX
@inproceedings{liang2019strong,
  author = {Paul Pu Liang and Yao Chong Lim and Yao-Hung Hubert Tsai and Ruslan Salakhutdinov and Louis-Philippe Morency},
  title = {{Strong and Simple Baselines for Multimodal Utterance Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1267},
}
Figure