Abstract
This paper challenges the prevailing trend of utilizing increasingly complex neural architectures for multimodal tasks. We introduce two simple, linear-based baselines for learning multimodal utterance embeddings: Multimodal Baseline 1 (MMB1) and MMB2. These models leverage weighted averages of features and closed-form solutions, achieving competitive performance with state-of-the-art deep learning models. Notably, our approach is over 10 times faster during inference and demonstrates superior robustness in low-data scenarios, suggesting that simple linear models can provide a powerful foundation for multimodal learning.
Introduction
Human communication is inherently multimodal, integrating text, visual cues (facial expressions, gestures), and acoustic signals (tone of voice). To enable machines to understand utterances, researchers strive to learn joint representations, or embeddings, that capture these diverse signals. Recent trends in the field have favored complex "black box" models, such as LSTMs, GRUs, and attention mechanisms. While effective, these models are computationally intensive, data-hungry, and often lack transparency. This research investigates whether such complexity is necessary or if simple linear models can serve as strong baselines.
Methodology
We propose two primary models: Multimodal Baseline 1 (MMB1) and MMB2.
MMB1, the Factorized Unimodal Model, assumes that the multimodal signal can be decomposed into independent unimodal factors conditioned on a joint embedding. Language is modeled using a log-linear word production model, while visual and acoustic signals are modeled as being generated from Gaussian distributions. A key mathematical contribution is the proof that the optimal joint embedding can be solved in closed form, resulting in a weighted average of features where weights are determined by word frequency and signal variance.
MMB2 extends this by incorporating "early fusion" factors to capture interactions between modalities (e.g., bimodal and trimodal interactions), maintaining the efficient linear framework. Optimization is performed via a coordinate-ascent style algorithm, alternating between the closed-form embedding solution and gradient-based updates for linear transformation parameters.
Experimental Evaluation
The models were evaluated on two benchmark datasets: CMU-MOSI for multimodal sentiment analysis and POM (Persuasive Opinion Multimedia) for speaker trait recognition. These datasets provide a rigorous testing ground for sentiment and personality trait detection from video clips.
Results and Discussion
Our simple baselines demonstrated several key advantages over complex neural counterparts:
- Competitive Performance: MMB2 matched or exceeded the accuracy of complex models like the Memory Fusion Network (MFN) and Tensor Fusion Network (TFN).
- Efficiency: MMB1 and MMB2 achieved massive speedups, ranging from 10x to 100x in inferences per second (IPS). For instance, MMB1 processed 421,000 IPS compared to MFN's 4,760.
- Robustness to Data Scarcity: In semi-supervised settings with only 40-60% of labels, our low-parameter baselines significantly outperformed deep autoencoders, which are prone to overfitting.
- Distributional Assumptions: Empirical evidence supports the use of Gaussian likelihoods for non-verbal signals. Histograms of pre-extracted visual and acoustic features show they closely follow Gaussian distributions over time.
Conclusion
This work demonstrates that simple linear models, when properly formulated, can serve as exceptionally strong baselines for multimodal utterance embeddings. By prioritizing efficiency and transparency, these models offer a viable alternative to complex neural networks, particularly in resource-constrained or low-data environments. We suggest that these baselines should be the standard starting point for future multimodal research to ensure that added complexity provides genuine incremental value.
Summary updated