TL;DR
This paper introduces a more efficient approach to multimodal sequence understanding by using low-rank fusion to combine information from different modalities (vision, audio, text) before feeding it into Transformer models. This method achieves comparable performance to existing, more complex models while significantly reducing computational cost and training time.
What Problem Does This Paper Address?
Understanding human emotions and sentiments often requires integrating information from multiple sensory channels – what we see, hear, and read. This is particularly challenging because these signals are often asynchronous and have different temporal dynamics. Existing methods, like the Multimodal Transformer (MulT), use numerous parallel Transformer models to capture these cross-modal interactions. However, this leads to significant computational overhead and long training times. This paper aims to address this by developing a more parsimonious architecture that can effectively fuse multimodal information without the need for an excessive number of complex cross-modal attention mechanisms.
Methods
The core innovation lies in the use of Low Rank Matrix Factorization (LMF) for multimodal fusion. Instead of directly combining high-dimensional embeddings from different modalities, LMF approximates the multimodal interaction by factorizing the interaction space into lower-dimensional "modality-specific factors." This allows for capturing unimodal, bimodal, and trimodal interactions more efficiently.
The authors then integrate this LMF fusion into a Transformer architecture. They propose two main approaches:
- LMF-MulT: The fused multimodal signal from LMF is then processed by a Transformer.
- Fusion-Based-CM-Attn: Individual modalities are first processed by their own Transformers and then attend to the fused signal in parallel.
Both methods leverage cross-modal attention, but in a more streamlined manner compared to previous work. Temporal convolutions are also used after LMF to help the model capture temporal dependencies.
Datasets
The proposed models were evaluated on three standard multimodal datasets:
- CMU-MOSI: A dataset of YouTube videos with sentiment annotations.
- CMU-MOSEI: A larger-scale dataset of YouTube videos with sentiment and emotion annotations.
- IEMOCAP: A dataset containing acted and spontaneous emotional speech.
The authors used both aligned and unaligned sequences from these datasets, highlighting the robustness of their approach to temporal discrepancies.
Key Insights and Ideas
- Low-Rank Fusion as an Efficiency Enabler: The key insight is that by using low-rank factorization, we can approximate complex multimodal interactions without the quadratic explosion of parameters and computations that arise from direct tensor products or numerous cross-modal attention heads.
- Streamlined Cross-Modal Attention: The paper demonstrates that effective multimodal understanding can be achieved with fewer cross-modal attention blocks by first fusing information at a lower rank and then allowing modalities to attend to this fused representation or vice-versa.
- Performance Without Over-Parameterization: The results suggest that achieving state-of-the-art performance doesn't necessarily require the most complex architectures. Efficiency gains can be realized without sacrificing accuracy.
Key Results
- Comparable Performance: The proposed LMF-MulT and Fusion-Based-CM-Attn models achieve performance comparable to the original MulT model on sentiment analysis and emotion recognition tasks across CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets.
- Significant Speedup: The paper reports a consistent speedup in training time for their models compared to MulT, as evidenced by the "Average Time/Epoch" data in Table 4. For instance, LMF-MulT is significantly faster than MulT on all datasets.
- Reduced Parameter Count: The proposed models use substantially fewer trainable parameters than the MulT model (Table 5), making them more memory-efficient and easier to deploy. For example, LMF-MulT uses roughly half the parameters of MulT on CMU-MOSI.
Limitations and Future Work
The authors acknowledge that their current results do not consistently outperform the original MulT model, and they plan to conduct more extensive hyperparameter tuning to further optimize their models.
Future work directions include:
- Exploring methods to compress temporal sequences without relying on LSTM hidden states, which can lose temporal information. They propose using convolution layers for this.
- Investigating deployment in low-resource settings with further optimizations.
- Exploring the use of richer features for audio, text, and vision pipelines for other applications.
Summary updated