DQ-Former: Querying Transformer with Dynamic Modality Priority for Cognitive-aligned Multimodal Emotion Recognition in Conversation

Jing Ye, Xinpei ZhaoACM Multimedia 2024 · OpenReview

TL;DR

This paper introduces DQ-Former, a novel multimodal fusion framework for emotion recognition in conversations. It addresses the limitations of previous methods by dynamically prioritizing different modalities and their granular emotional cues across intermediate layers, leading to more accurate and interpretable emotion detection.

What Problem Does This Paper Address?

Recognizing emotions in conversations is crucial for natural human-computer interaction. While existing methods often rely on complex unimodal feature extraction or generic fusion techniques, they overlook how humans perceive emotions. Human emotions are conveyed not just through semantics but also through nuanced, modality-specific cues (like pitch in speech or word choice in text), and the importance of these cues can vary significantly from one utterance to another. Previous approaches often fail to capture these varying levels of emotional information and the dynamic interplay between modalities, leading to suboptimal performance. This paper aims to bridge this gap by proposing a framework that mimics human cognitive processes more closely.

Methods

The core of DQ-Former is a middle-fusion approach that operates on intermediate representations from pre-trained unimodal encoders (like BERT for text and WavLM for audio). Instead of fusing only at the end (late-fusion) or combining raw inputs (early-fusion), DQ-Former uses a small set of learnable query tokens. These tokens act as "information gatherers," interacting with the intermediate layers of the unimodal encoders through cross-modal attention.

The key innovations are:

  • Query-based Collation: Learnable query tokens condense granular emotion cues from different layers of unimodal encoders.
  • Dynamic Modality Priority (DMP): A novel module that learns to dynamically assign importance to each modality for every specific utterance. This is achieved through a "ConNet" that estimates confidence scores for each modality's contribution. A mask threshold is used to filter out potentially misleading signals.
  • Middle-Fusion: Fusion occurs across multiple intermediate layers, allowing the model to leverage diverse levels of emotional information (e.g., semantic features in higher layers, acoustic nuances in intermediate layers).

Datasets

The proposed DQ-Former model was evaluated on two widely recognized multimodal emotion recognition benchmark datasets:

  • IEMOCAP: An interactive emotional dyadic motion capture database.
  • MELD: A multimodal multi-party dataset collected from the Friends TV show.

The paper also mentions using a subset of IEMOCAP (4-way condition) for specific ablation studies.

Key Insights and Ideas

  • Human Perception Matters: The paper strongly advocates for a cognitive-aligned approach, recognizing that human emotion perception is dynamic and context-dependent, not a one-size-fits-all process.
  • Intermediate Layers Hold Valuable Cues: Unlike late-fusion methods that only use final representations, DQ-Former's middle-fusion strategy effectively taps into the rich, hierarchical information present in the intermediate layers of pre-trained unimodal models.
  • Modality Importance is Dynamic: The most significant insight is that the "best" modality for emotion recognition isn't fixed. Sometimes text is dominant, other times audio is. DQ-Former's Dynamic Modality Priority (DMP) module explicitly models and adapts to this dynamic importance at an instance level.
  • Filtering Noise is Crucial: The DMP module's ability to mask or down-weight less informative or misleading modalities is key to achieving robust performance, especially in noisy or ambiguous situations.

Key Results

DQ-Former demonstrated state-of-the-art or competitive performance on both IEMOCAP and MELD datasets.

  • Overall Performance: DQ-Former achieved significant improvements over established late-fusion (MulT) and middle-fusion (MBT) methods. For instance, it showed a 2.91% and 6.51% WF1 improvement over MulT on MELD and IEMOCAP, respectively.
  • Robustness: The model showed superior robustness to modality-missing scenarios compared to other multimodal models, indicating it doesn't overly rely on a single modality.
  • Interpretability: The confidence scores generated by the DMP module provided insights into modality prioritization across layers and instances, aligning with human cognitive tendencies. For example, text modality was found to be more influential in lower layers, while acoustic modality gained prominence in middle layers.

Limitations and Future Work

  • Model Size vs. Data: The paper notes that the "large" version of DQ-Former did not always outperform the "base" version, potentially due to limited existing data and the high parameter count leading to overfitting.
  • Mask Threshold Sensitivity: While the DMP module is effective, the paper suggests that carefully selecting the mask threshold is important, as an overly high threshold might filter out useful signals.
  • Future Work: The authors suggest further research into leveraging advanced modality-specific pre-trained models from a cognitive perspective to improve multimodal emotion recognition. The interpretability findings also open avenues for more nuanced understanding of how different modalities contribute to emotion recognition.

Summary updated

Abstract

Multimodal Emotion Recognition in Conversations aims to understand the human emotion of each utterance in a conversation from different types of data, such as speech and text. Previous works mainly focus on either complex unimodal feature extraction or sophisticated fusion techniques as general multimodal classification tasks do. However, they ignore the process of human perception, neglecting various levels of emotional features within each modality and disregarding the unique contributions of different modalities for emotion recognition. To address these issues, we propose a more cognitive-aligned multimodal fusion framework, namely DQ-Former. Specifically, DQ-Former utilizes a small set of learnable query tokens to collate and condense various granularities of emotion cues embedded at different layers of pre-trained unimodal models. Subsequently, it integrates these emotional features from different modalities with dynamic modality priorities at each intermediate fusion layer. This process enables explicit and effective fusion of different levels of information from diverse modalities. Extensive experiments on MELD and IEMOCAP datasets validate the effectiveness of DQ-Former. Our results show that the proposed method achieves a robust and interpretable multimodal representation for emotion recognition.
BibTeX
@inproceedings{ye2024dq,
  author = {Jing Ye and Xinpei Zhao},
  title = {{DQ-Former: Querying Transformer with Dynamic Modality Priority for Cognitive-aligned Multimodal Emotion Recognition in Conversation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=VyqQtIg5SH},
}
Figure