Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

Dong Zhang, Xincheng Ju, Wei Zhang, Junhui Li, Shoushan Li, Qiaoming Zhu, Guodong ZhouAAAI 2021 · OpenReview

Abstract

As an important research issue in the affective computing community, multi-modal emotion recognition has become a hot topic in the last few years. However, almost all existing studies perform multiple binary classification for each emotion with a focus on complete time series data. In this paper, we focus on multi-modal emotion recognition in a multi-label scenario. In this scenario, we consider not only the label-to-label dependency, but also the feature-to-label and modality-to-label dependencies. Particularly, we propose a heterogeneous hierarchical message passing network to effectively model these dependencies. Furthermore, we propose a new multi-modal multi-label emotion dataset based on partial time-series content to show the predominant generalization of our model. Detailed evaluation demonstrates the effectiveness of our approach.

Introduction

Multi-modal emotion recognition (MER) is a critical task in building systems that can interpret human affect through various channels such as text, audio, and visual cues. Despite its importance, current research often relies on two restrictive assumptions. First, many models treat emotions as independent binary tasks or assume a single dominant emotion, failing to capture the nuanced reality where a person may feel a mix of emotions simultaneously (e.g., "happiness" and "surprise"). Second, existing models typically rely on sequential processing architectures like LSTMs or Transformers, which assume data flows in a strict chronological order. This is problematic for real-world multimedia content, such as social media comments or music metadata, which often lacks a continuous temporal flow.

To address these limitations, we introduce the Heterogeneous Hierarchical Message Passing Network (HHMPN). This framework treats features and labels as nodes in a graph, allowing for the modeling of complex dependencies without being constrained by sequential processing. This approach enables the system to handle both traditional sequential data and "partial time-series" multimedia content effectively.

Multi-modal inputs for (a) sequential video data and (b) non-sequential music metadata, illustrating the multi-label emotion recognition task.

Methodology: Heterogeneous Hierarchical Message Passing

The core innovation of the HHMPN is its use of a graph-based approach to "pass messages" between different types of information. Instead of a flat architecture, it operates across four hierarchical levels to capture the intricate web of dependencies in emotional expression:

  • Feature-to-Feature (F2F): This level models how different modalities interact. For example, it can learn how a specific tone of voice changes the perceived meaning of a spoken word.
  • Feature-to-Label (F2L): This module learns which specific features, such as a facial grimace or a specific keyword, contribute most directly to specific emotion labels.
  • Label-to-Label (L2L): This captures the statistical correlation between emotions. If the model detects "sadness," the L2L module recognizes a higher probability that "lonely" is also present, reflecting natural emotional co-occurrence.
  • Modality-to-Label (M2L): This adaptively weights the importance of each modality. For some emotions, lyrics might be the primary indicator, while for others, the acoustic melody or cover art may be more telling.

By utilizing Message Passing Neural Networks (MPNNs) with self-attention, the model dynamically learns these relationships without requiring a fixed, predefined graph structure, making it both modality-agnostic and order-agnostic.

Datasets

The authors evaluated the HHMPN on two distinct benchmarks to test its versatility:

  1. CMU-MOSEI: The standard large-scale dataset for multi-modal emotion recognition, consisting of video utterances with aligned text, visual, and acoustic data.
  2. NEMu (NetEase Cloud Music): A newly introduced dataset featuring "partial time-series" data. NEMu includes lyrics, independent user comments, audio, and cover art. This dataset is specifically designed to test the model's ability to generalize to non-sequential, real-world multimedia content where traditional sequential models often struggle.

Key Results and Insights

Experimental results show that the HHMPN consistently outperforms state-of-the-art (SOTA) models, including the Multimodal Transformer (MulT) and M3ER:

  • Superior Performance: On the CMU-MOSEI dataset, HHMPN achieved a Micro F1-score of 55.6%, significantly higher than MulT's 53.1%. The advantage was even more pronounced on the NEMu dataset, where HHMPN reached a Micro F1 of 46.1% compared to MulT’s 42.6%.
  • Cross-Modal Synergy: Ablation studies revealed that the Feature-to-Feature (F2F) module is the most critical component. Removing it caused the largest drop in performance, proving that understanding how different senses interact is vital for accurate emotion recognition.
  • Robustness: The model maintained high performance even when one or more modalities (e.g., video or audio) were missing. This "modality-aware" reasoning allows the network to compensate for incomplete data by leveraging the dependencies learned across the remaining features and labels.

Conclusion and Future Work

The HHMPN framework successfully addresses the complexities of multi-modal, multi-label emotion recognition by moving beyond sequential constraints. While the model performs exceptionally well on common emotions, it faces challenges with the "long-tail" distribution of labels, struggling with low-frequency emotions like "Excited" or "Relaxed." Future work will explore specialized loss functions and the integration of external knowledge bases, such as emotion lexicons, to further refine label dependencies and improve performance on rare emotional states.

Summary updated

Abstract

As an important research issue in affective computing community, multi-modal emotion recognition has become a hot topic in the last few years. However, almost all existing studies perform multiple binary classification for each emotion with focus on complete time series data. In this paper, we focus on multi-modal emotion recognition in a multi-label scenario. In this scenario, we consider not only the label-to-label dependency, but also the feature-to-label and modality-to-label dependencies. Particularly, we propose a heterogeneous hierarchical message passing network to effectively model above dependencies. Furthermore, we propose a new multi-modal multi-label emotion dataset based on partial time-series content to show predominant generalization of our model. Detailed evaluation demonstrates the effectiveness of our approach.
BibTeX
@inproceedings{zhang2021multi,
  author = {Dong Zhang and Xincheng Ju and Wei Zhang and Junhui Li and Shoushan Li and Qiaoming Zhu and Guodong Zhou},
  title = {{Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Xycux1hSXN},
}
Figure