Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Zhexian Huang, Bo Zhao, Hui Ma, Zhishu Liu, Jie Zhang, Ruixin Zhang, Shouhong Ding, Zitong YuarXiv 2026 · arXiv

TL;DR

The paper introduces Atsuko, a novel multimodal emotion recognition framework that decomposes audio, visual, and text signals into fine-grained, orthogonal spectral bands to capture subtle emotional cues. By leveraging a game-theoretic approach to calculate the true marginal contribution of each modality, Atsuko eliminates reliance on dominant modalities and achieves state-of-the-art performance across five major benchmarks.

What Problem Does This Paper Address?

Multimodal Emotion Recognition (MER) aims to predict human emotions by fusing text, video, and audio. Despite recent advances, current MER models suffer from two fundamental flaws:

  1. Coarse-Grained Fusion: Existing methods treat temporal features as homogeneous signals. This discards fine-grained emotional cues that concentrate within specific frequency bands—such as rapid, high-frequency facial micro-expressions or slow, low-frequency vocal baselines.
  2. Shortcut Learning: Models often over-rely on a single "dominant" modality (typically text) because it yields strong independent performance. However, a seemingly weaker modality (like audio or video) might contain critical, non-redundant information that is vital for accurate multimodal prediction.
Figure 1: Two fundamental limitations of existing MER methods. (a) Coarse-grained fusion: treating each modality’s temporal features as a homogeneous signal discards fine-grained emotional cues in specific frequency bands. (b) Shortcut learning: performance-based weighting over-relies on the dominant (text) modality, whereas visual and acoustic modalities carry the genuinely complementary cues. Complementarity-based weighting correctly amplifies these contributions.

To resolve these issues, the authors propose the Complementarity-Supervised Multi-Band Expert Network, nicknamed Atsuko.

Methods

Atsuko introduces a hierarchical approach that operates at both the spectral-band level and the modality level.

The overall architecture of Atsuko, highlighting spectral-band routing, modality-level routing, and the Marginal Complementarity Module.

1. Spectral-Band Hierarchical Routing Network

Instead of using standard Fast Fourier Transforms (FFT)—which assume infinite periodicity and ignore local temporal boundaries—Atsuko applies graph Laplacian eigen-decomposition to model the finite-length temporal sequences.

  • Equal-Energy Partitioning: Because low-frequency components hold most of the signal's energy, standard partitioning schemes often ignore high-frequency details. Atsuko calculates a data-driven boundary map to ensure each sub-band (high, mid, low) contains equal signal energy.
  • Band Router & Masking: A lightweight router dynamically weights these bands per sample. During training, Stochastic Band Masking randomly drops bands, forcing the network to find alternative emotional cues and boosting noise robustness.

2. Modality-Level Router

A dual-path mechanism allows the model to route signals using either a Pre-Attention Path (decoupling routing from deep feature extraction) or a Post-Attention Path (capturing deep semantic summaries requiring contextual reasoning).

3. Marginal Complementarity Module (MCM)

Inspired by the Shapley value concept from cooperative game theory, the MCM evaluates three parallel bimodal branches (e.g., predicting emotion without text, without vision, or without audio). By measuring the performance drop when a specific modality is removed, the system quantifies that modality's true marginal contribution. This distribution acts as a soft supervision signal (via KL divergence minimization) to guide the modality router away from shortcut learning.

Datasets

The framework was evaluated on five public benchmarks:

  • CMU-MOSI & CMU-MOSEI: Standard English-language multimodal sentiment and emotion analysis datasets, tested under both word-aligned and unaligned settings.
  • CH-SIMS & CH-SIMSv2: Chinese multimodal datasets featuring independent, fine-grained annotations for each modality.
  • MIntRec: A multimodal intent recognition dataset used to validate the model's cross-task generalization capability.

Key Insights and Ideas

  • Unimodal Performance $\neq$ Multimodal Value: A modality that performs poorly on its own can still provide the most critical incremental information when integrated into a multimodal system. MCM successfully forces the model to prioritize these unique information gains over redundant independent performance.
  • Frequency Specialization: Visualization reveals that the router adaptively targets different frequencies based on the modality. For text, it prioritizes ultra-low frequencies (global context). For audio and vision, it focuses heavily on mid-to-high frequencies to capture prosodic fluctuations and dynamic facial muscle movements.
  • Orthogonal Decoupling: Decomposing features into mathematically orthogonal bands ensures zero information loss during spectral analysis, allowing the band router to evaluate components independently.

Key Results

Atsuko consistently outperformed all baseline models (including recent advanced architectures like EMOE, Self-MM, and MISA) across all five datasets.

Performance comparison on CMU-MOSI and CMU-MOSEI datasets under aligned and unaligned settings.
  • CMU-MOSI (Unaligned): Achieved a top Acc-7 of 49.4% (outperforming the previous best method, EMOE, by 1.6%) and achieved the lowest Mean Absolute Error (MAE) of 0.691.
  • CH-SIMS: Improved the 5-class accuracy (Acc-5) metric by 2.19 percentage points over EMOE, demonstrating strong cross-lingual performance.
  • MIntRec: Achieved the highest accuracy (74.16%) and F1 score (71.22%) on intent recognition, proving that the routing mechanism generalizes effectively beyond standard emotion tasks.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states. Prior methods face two main limitations: mechanically relying on independent unimodal performance, thereby missing genuine complementary contributions, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks. As inconsistent information density across heterogeneous modalities hinders inter-modal feature mining, we propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration. Specifically, we orthogonally decompose each modality's features into high, mid, and low-frequency components. Building upon this band-level routing, we design a modality-level router with a dual-path mechanism for fine-grained cross-band selection and cross-modal fusion. To mitigate shortcut learning from dominant modalities, we propose the Marginal Complementarity Module (MCM) to quantify performance loss when removing each modality via bi-modal comparison. The resulting complementarity distribution provides soft supervision, guiding the router to focus on modalities contributing unique information gains. Extensive experiments show our method achieves superior performance on the CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2, and MIntRec benchmarks.
BibTeX
@misc{huang2026complementarity,
  author = {Zhexian Huang and Bo Zhao and Hui Ma and Zhishu Liu and Jie Zhang and Ruixin Zhang and Shouhong Ding and Zitong Yu},
  title = {{Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition}},
  year = {2026},
  eprint = {2603.13340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.13340},
}
Figure