Abstract: Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.
Introduction
In recent years, multimodal learning has been dominated by increasingly complex architectures. Researchers have leaned heavily on Recurrent Neural Networks (RNNs), LSTMs, attention mechanisms, and memory components to fuse heterogeneous data streams like speech, facial expressions, and text. While these models perform well, they come with high computational costs and long inference times. This paper addresses the lack of strong, simple baselines in the field, proving that fundamental statistical assumptions can rival deep learning complexity.
Methodology
The authors introduce two non-neural models: Multimodal Baseline 1 (MMB1) and MMB2.
- MMB1 (Factorized Unimodal Model): This model assumes that a multimodal utterance can be factorized into independent unimodal factors (language, visual, and acoustic) conditioned on a joint embedding. Language is modeled using a log-linear word production model, while visual and acoustic features are modeled using Gaussian likelihood functions. The authors prove that the maximum likelihood estimate for the joint embedding can be derived in closed form as a weighted average of the features.
- MMB2 (Bimodal and Trimodal Interactions): This extends MMB1 by adding factors for interactions between modalities (e.g., how a specific facial expression changes the meaning of a spoken word) using "early fusion" via feature concatenation.
To handle temporal information without RNNs, the models incorporate positional encodings. Optimization is performed using a coordinate-ascent style algorithm that alternates between solving for the optimal embedding and updating linear transformation parameters via gradients.
Experimental Results
The models were evaluated on two benchmarks: CMU-MOSI (multimodal sentiment analysis) and POM (speaker trait recognition).
Competitive Accuracy
On the CMU-MOSI sentiment task, MMB2 achieved an accuracy of 75.2%, outperforming several complex neural models like TFN (74.6%) and BC-LSTM (73.9%). Similar competitive performance was observed across the personality trait recognition tasks in the POM dataset.
Efficiency and Robustness
MMB1 and MMB2 are 10 to 100 times faster during inference than neural counterparts. For instance, MMB1 can process 421,000 inferences per second, compared to just 1,400 for a standard Multi-View LSTM. Furthermore, these baselines are highly robust in low-data regimes; in semi-supervised tests, MMB2 maintained high accuracy even when 60% of labels were removed, significantly outperforming deep autoencoders.
Discussion and Conclusion
This research serves as a "reality check" for the multimodal community. It demonstrates that much of the multimodal signal can be captured by linear transformations and weighted averages. While the closed-form solution relies on certain assumptions, such as the optimal embedding lying on a unit sphere, the resulting models offer a powerful, efficient alternative to black-box neural architectures. Future work may explore more nuanced ways of modeling cross-modal dynamics within this simple framework.
Summary updated