Low Rank Fusion based Transformers for Multimodal Sequences

Saurav Sahay, Eda Okur, Shachi H Kumar, Lama Nachman

TL;DR

This paper introduces a more efficient way to handle multimodal data (like video, audio, and text) by using Low Rank Matrix Factorization (LMF) to fuse signals before passing them through Transformers. By replacing complex, over-parameterized cross-modal architectures with a central "fused" representation, the authors achieved comparable performance to state-of-the-art models while significantly reducing parameter counts and training time.

What Problem Does This Paper Address?

Human communication is inherently multimodal: we use words, tone of voice, and facial expressions simultaneously to convey emotion. In machine learning, capturing the "interplay" between these different data streams is challenging.

Previous state-of-the-art models, like the Multimodal Transformer (MulT), attempt to solve this by using a massive array of parallel Transformers—at least nine for a three-modality setup—to attend to every possible pair of interactions (e.g., Audio-to-Text, Text-to-Vision, etc.). This leads to two main issues:

  1. Over-parameterization: The models become extremely large and slow to train.
  2. Missing Trimodal Interplay: These models often fail to capture the simultaneous interaction of all three modalities in a single representation.

Methods

The authors propose a streamlined architecture that centers on a fused multimodal signal rather than exhaustive pairwise comparisons.

1. Low Rank Matrix Factorization (LMF)

Instead of using a computationally expensive "tensor product" to combine modalities (which grows exponentially with the number of inputs), the authors use LMF. This technique approximates the complex interactions between vision, audio, and text using a smaller set of modality-specific factors. This captures unimodal, bimodal, and trimodal interactions efficiently.

Figure 2: Low Rank Matrix Factorization

2. Two Novel Architectures

The researchers experimented with two ways to integrate this fused signal with Transformers:

  • LMF-MulT: The individual modalities (Vision, Audio, Text) attend to the central fused signal in parallel. This uses the fewest number of Transformer layers.
  • Fusion-Based-CM-Attn: The fused multimodal signal is reinforced by attending to each of the three individual modalities.
Figure 3: Fused Cross-modal Transformer

To ensure the model understands the sequence of events, they use LSTMs for initial compression and 1D Convolutions after the fusion step to recover temporal context.

Datasets

The models were evaluated on three standard benchmarks for multimodal sentiment and emotion recognition:

  • CMU-MOSI: A dataset of YouTube movie reviews with sentiment annotations.
  • CMU-MOSEI: A larger-scale version of MOSI with over 23,500 sentence utterances.
  • IEMOCAP: A dataset of scripted and improvised dialogues annotated for specific emotions (Happy, Sad, Angry, Neutral).

The authors tested their models on both aligned sequences (where signals are synchronized in time) and unaligned sequences, proving the robustness of their attention-based approach.

Key Insights and Ideas

  • Efficiency as a First-Class Citizen: You don't need a Transformer for every possible modality pair. A central "fused hub" can act as a proxy for alignment and interaction, drastically reducing the model's footprint.
  • Trimodal Capture: By using LMF, the model explicitly accounts for interactions between all three senses at once, which pairwise models might miss.
  • Attention as Alignment: The cross-modal attention mechanism acts as a proxy for signal synchronization. This removes the need for manual, time-consuming data alignment before training.

Key Results

The proposed models achieved performance comparable to the much larger MulT model while being significantly leaner:

  • Parameter Reduction: On the CMU-MOSEI dataset, the Fusion-Based-CM-Attn model used roughly 531k parameters, compared to the 1.07M parameters required by the standard MulT—a reduction of nearly 50%.
  • Faster Training: The LMF-MulT model was the fastest, showing a consistent speedup in training time per epoch across all datasets. For example, on CMU-MOSEI, it took ~137 seconds per epoch compared to MulT's ~216 seconds.
  • Competitive Accuracy: Despite having fewer parameters, the models often matched or slightly exceeded SOTA results, particularly in emotion recognition tasks on the IEMOCAP dataset.

Limitations and Future Work

The authors noted a few areas for improvement:

  • Hyperparameter Tuning: The current results were achieved without exhaustive grid searches for optimal settings; doing so might further bridge the performance gap with larger models.
  • Temporal Compression: The initial use of LSTMs to compress sequences can lead to a loss of temporal information. The authors are exploring ways to compress sequences without relying on the hidden states of LSTMs.
  • Feature Richness: Future work will involve using more sophisticated features for the individual audio, text, and vision pipelines to see if the efficiency gains hold at higher levels of data complexity.

Summary updated

Abstract

Our senses individually work in a coordinated fashion to express our emotional intentions. In this work, we experiment with modeling modality-specific sensory signals to attend to our latent multimodal emotional intentions and vice versa expressed via low-rank multimodal fusion and multimodal transformers. The low-rank factorization of multimodal fusion amongst the modalities helps represent approximate multiplicative latent signal interactions. Motivated by the work of~\cite{tsai2019MULT} and~\cite{Liu_2018}, we present our transformer-based cross-fusion architecture without any over-parameterization of the model. The low-rank fusion helps represent the latent signal interactions while the modality-specific attention helps focus on relevant parts of the signal. We present two methods for the Multimodal Sentiment and Emotion Recognition results on CMU-MOSEI, CMU-MOSI, and IEMOCAP datasets and show that our models have lesser parameters, train faster and perform comparably to many larger fusion-based architectures.
BibTeX
@inproceedings{sahay2020low,
  author = {Saurav Sahay and Eda Okur and Shachi H Kumar and Lama Nachman},
  title = {{Low Rank Fusion based Transformers for Multimodal Sequences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}

Loading related content...

Figure