TL;DR
This paper introduces TSDA, a novel framework that disentangles temporal and spatial information before multimodal fusion for sentiment analysis. By explicitly separating these factors, TSDA mitigates "spatiotemporal asymmetry" and achieves state-of-the-art performance on challenging multimodal sentiment analysis benchmarks.
What Problem Does This Paper Address?
Multimodal Sentiment Analysis (MSA) aims to understand human emotions by integrating information from various sources like text, audio, and video. While existing methods often fuse these modalities early on, they tend to struggle with "spatiotemporal information asymmetry." This means that the model might overemphasize static, easily detectable cues (like the speaker's identity or sentence structure) while neglecting crucial, but often fleeting, temporal dynamics (like prosodic changes or micro-expressions) that are vital for accurate sentiment prediction. This early entanglement of temporal and spatial information leads to models that are brittle and perform suboptimally. The paper argues that this asymmetry is a key bottleneck preventing further progress in MSA.
Methods
The core innovation of TSDA lies in its "Temporal-Spatial Decouple before Act" approach. Here's a breakdown of the key components:
- Temporal and Spatial Disentanglement: For each modality (language, visual, acoustic), TSDA employs separate temporal and spatial encoders. The temporal encoder captures the sequential nature of the data (e.g., word order, video frames over time), while the spatial encoder extracts time-invariant structural context (e.g., sentence meaning, visual scene layout). This creates two distinct "bodies" of representation for each modality.
- Factor-Consistent Cross-Modal Alignment (FCCA): After disentanglement, TSDA aligns these factor-specific representations across modalities. Crucially, it uses a "block-diagonal masked attention" mechanism. This ensures that temporal features from one modality are only aligned with temporal features from other modalities, and similarly for spatial features. This prevents cross-factor interference.
- Factor Purity and Decorrelation Regularization: To further enforce the separation and prevent leakage between temporal and spatial streams, TSDA uses two regularization techniques:
- Factor Purity Supervision: A discriminator is trained to predict whether an aligned token originates from the temporal or spatial stream, penalizing the model if it misclassifies.
- Decorrelation Regularization: This penalizes second-order and nonlinear dependencies between the aligned temporal and spatial summaries, encouraging them to be complementary rather than redundant.
- Gated Recouple Module (GR): Finally, the aligned temporal and spatial summaries are adaptively fused. The GR module uses instance-wise calibrated weights, informed by factor disagreement and confidence signals, to dynamically balance the contributions of the temporal and spatial streams. This helps mitigate static dominance by allowing the model to rely more on temporal cues when they are more informative.
Datasets
TSDA was evaluated on two standard multimodal sentiment analysis benchmarks:
- CMU-MOSI: Contains 2,199 video segments with aligned text, audio, and visual data.
- CMU-MOSEI: A larger dataset with 23,453 clips from 1,000 speakers, also featuring aligned text, audio, and visual information.
Both datasets have sentiment labels ranging from -3 to +3.
Key Insights and Ideas
- The critical role of temporal dynamics: The paper's ablation studies reveal that removing the temporal stream leads to a sharper performance drop than removing the spatial stream. This strongly supports the hypothesis that fleeting temporal cues are often more decisive for sentiment than static context.
- Early disentanglement is key: The success of TSDA hinges on separating temporal and spatial factors before any cross-modal interaction. This proactive approach prevents the entanglement that plagues many existing methods.
- Factor-consistent alignment prevents interference: The block-diagonal attention mechanism in FCCA is crucial for ensuring that temporal and spatial information are aligned independently, thereby avoiding contamination.
- Adaptive fusion is more than just weighting: While simple fusion methods like concatenation or summation are outperformed, the gated recouple module's ability to adaptively balance streams based on reliability signals is vital, but it works best in conjunction with the disentangled and aligned representations.
- Regularization is essential for clean factors: The purity and decorrelation regularizers are not just add-ons; they are central to enforcing the desired factor separation and complementarity, leading to significant performance gains.
Key Results
TSDA achieved state-of-the-art performance on both CMU-MOSI and CMU-MOSEI datasets across multiple metrics, including MAE (Mean Absolute Error), ACC-7 (7-class accuracy), ACC-2 (2-class accuracy), and F1-score.
- On CMU-MOSI, TSDA reduced MAE from 0.710/0.697 to 0.695/0.680 and improved ACC7, ACC2, and F1 by approximately 1% compared to the strongest baseline (EMOE).
- On CMU-MOSEI, TSDA achieved the lowest MAE (0.529/0.527) and the highest accuracy and F1 scores.
- Qualitative analysis using t-SNE visualizations demonstrated that TSDA produces more compact and monotonic sentiment embeddings, indicating better disentanglement and alignment compared to baseline methods.
- Ablation studies confirmed the necessity of each component, with the removal of language modality causing the largest performance degradation, highlighting its importance in sentiment analysis.
Limitations and Future Work
- Limitations: The paper acknowledges that while the gated recouple module helps, minor cross-factor leakage can still persist in some scenarios. The sensitivity analysis shows that while TSDA is relatively robust to hyperparameter choices, optimal values still exist.
- Future Work: The authors plan to extend the TSDA framework to the field of human-computer interaction. Further research could explore more sophisticated disentanglement techniques or investigate how TSDA can be applied to other multimodal tasks beyond sentiment analysis.
Summary updated