TL;DR
The paper introduces Atsuko, a novel multimodal emotion recognition framework that decomposes audio, visual, and text signals into fine-grained, orthogonal spectral bands to capture subtle emotional cues. By leveraging a game-theoretic approach to calculate the true marginal contribution of each modality, Atsuko eliminates reliance on dominant modalities and achieves state-of-the-art performance across five major benchmarks.
What Problem Does This Paper Address?
Multimodal Emotion Recognition (MER) aims to predict human emotions by fusing text, video, and audio. Despite recent advances, current MER models suffer from two fundamental flaws:
- Coarse-Grained Fusion: Existing methods treat temporal features as homogeneous signals. This discards fine-grained emotional cues that concentrate within specific frequency bands—such as rapid, high-frequency facial micro-expressions or slow, low-frequency vocal baselines.
- Shortcut Learning: Models often over-rely on a single "dominant" modality (typically text) because it yields strong independent performance. However, a seemingly weaker modality (like audio or video) might contain critical, non-redundant information that is vital for accurate multimodal prediction.
To resolve these issues, the authors propose the Complementarity-Supervised Multi-Band Expert Network, nicknamed Atsuko.
Methods
Atsuko introduces a hierarchical approach that operates at both the spectral-band level and the modality level.
1. Spectral-Band Hierarchical Routing Network
Instead of using standard Fast Fourier Transforms (FFT)—which assume infinite periodicity and ignore local temporal boundaries—Atsuko applies graph Laplacian eigen-decomposition to model the finite-length temporal sequences.
- Equal-Energy Partitioning: Because low-frequency components hold most of the signal's energy, standard partitioning schemes often ignore high-frequency details. Atsuko calculates a data-driven boundary map to ensure each sub-band (high, mid, low) contains equal signal energy.
- Band Router & Masking: A lightweight router dynamically weights these bands per sample. During training, Stochastic Band Masking randomly drops bands, forcing the network to find alternative emotional cues and boosting noise robustness.
2. Modality-Level Router
A dual-path mechanism allows the model to route signals using either a Pre-Attention Path (decoupling routing from deep feature extraction) or a Post-Attention Path (capturing deep semantic summaries requiring contextual reasoning).
3. Marginal Complementarity Module (MCM)
Inspired by the Shapley value concept from cooperative game theory, the MCM evaluates three parallel bimodal branches (e.g., predicting emotion without text, without vision, or without audio). By measuring the performance drop when a specific modality is removed, the system quantifies that modality's true marginal contribution. This distribution acts as a soft supervision signal (via KL divergence minimization) to guide the modality router away from shortcut learning.
Datasets
The framework was evaluated on five public benchmarks:
- CMU-MOSI & CMU-MOSEI: Standard English-language multimodal sentiment and emotion analysis datasets, tested under both word-aligned and unaligned settings.
- CH-SIMS & CH-SIMSv2: Chinese multimodal datasets featuring independent, fine-grained annotations for each modality.
- MIntRec: A multimodal intent recognition dataset used to validate the model's cross-task generalization capability.
Key Insights and Ideas
- Unimodal Performance $\neq$ Multimodal Value: A modality that performs poorly on its own can still provide the most critical incremental information when integrated into a multimodal system. MCM successfully forces the model to prioritize these unique information gains over redundant independent performance.
- Frequency Specialization: Visualization reveals that the router adaptively targets different frequencies based on the modality. For text, it prioritizes ultra-low frequencies (global context). For audio and vision, it focuses heavily on mid-to-high frequencies to capture prosodic fluctuations and dynamic facial muscle movements.
- Orthogonal Decoupling: Decomposing features into mathematically orthogonal bands ensures zero information loss during spectral analysis, allowing the band router to evaluate components independently.
Key Results
Atsuko consistently outperformed all baseline models (including recent advanced architectures like EMOE, Self-MM, and MISA) across all five datasets.
- CMU-MOSI (Unaligned): Achieved a top Acc-7 of 49.4% (outperforming the previous best method, EMOE, by 1.6%) and achieved the lowest Mean Absolute Error (MAE) of 0.691.
- CH-SIMS: Improved the 5-class accuracy (Acc-5) metric by 2.19 percentage points over EMOE, demonstrating strong cross-lingual performance.
- MIntRec: Achieved the highest accuracy (74.16%) and F1 score (71.22%) on intent recognition, proving that the routing mechanism generalizes effectively beyond standard emotion tasks.
Limitations and Future Work
Not discussed in the paper.
Summary updated