TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection

Jiamin Luo, Jingjing Wang, Guodong ZhouCOLING 2024

TL;DR

This paper introduces TopicDiff, a novel approach for multimodal conversational emotion detection that leverages diffusion models to better capture topic information across acoustic, visual, and language modalities. By addressing the diversity deficiency in traditional topic models, TopicDiff significantly improves emotion detection accuracy, particularly highlighting the robustness of acoustic and visual topic information.

What Problem Does This Paper Address?

Conversational emotion detection (CED) is a crucial task for building empathetic AI systems, aiming to automatically identify emotions within conversations. While existing methods often focus on contextual information from various modalities (audio, video, text), they largely overlook the topic of the conversation. The authors argue that topics significantly influence emotions (e.g., funeral topics leading to sadness, wedding topics to happiness). Previous attempts to incorporate topic information have primarily relied on single modalities (usually text) and suffered from a "diversity deficiency" in capturing the full spectrum of topics. Furthermore, acoustic and visual modalities, which carry rich emotional cues, have been underutilized for topic extraction in CED. This paper aims to bridge this gap by proposing a method to effectively capture multimodal topic information for improved conversational emotion detection.

Methods

The core contribution is TopicDiff, a model-agnostic approach that integrates diffusion models into neural topic models. This integration is designed to overcome the "diversity deficiency" of traditional topic models, allowing for a richer representation of topics.

Here's a breakdown of the key components:

  • Topic-enriched Diffusion Block (TDB): This is the central novel component. It consists of two processes:
  • Topic-enriched Diffusion Process: This process perturbs latent topic representations with noise across multiple time scales, modeled by a stochastic differential equation (SDE). This allows for efficient generation of representations.
  • Topic-enriched Denoising Process: This process reverses the diffusion, synthesizing latent topic representations from a prior distribution. This is achieved by solving the reverse-time SDE.
  • Multimodal Topic Capture: TopicDiff employs three separate TDB modules to capture topic information from acoustic, vision, and language modalities independently.
  • Model-Agnostic Integration: TopicDiff is designed to be plug-and-play. It takes the context-aware representations from existing multimodal CED models and enriches them with multimodal topic information. The output is then fed into the final emotion prediction layer.
  • Joint Training: The basic MCE approach is trained jointly with the three TopicDiff modules using a multi-task learning objective that includes reconstruction loss and KL divergence loss to ensure accurate topic representation.

Datasets

The authors evaluate their approach on:

  • M3ED*: A newly constructed "topic-density" dataset derived from 56 different TV series. This dataset is designed to have richer topic variations compared to existing ones.
  • MELD: A public "topic-sparsity" dataset from the TV show Friends.
  • IEMOCAP: A public "topic-sparsity" dataset featuring dyadic conversations from actors.

The M3ED* dataset is notable for its topic richness, while MELD and IEMOCAP are used for comparison and to demonstrate TopicDiff's effectiveness on datasets with less explicit topic structure.

Key Insights and Ideas

  • Multimodal Topic Information is Crucial: The paper strongly advocates for the importance of topic information across all modalities (audio, vision, language) for accurate conversational emotion detection.
  • Diffusion Models Enhance Topic Modeling: Integrating diffusion models into neural topic models effectively addresses the "diversity deficiency" problem, leading to a more comprehensive capture of topic information.
  • Acoustic and Vision Topics are More Robust: A key finding is that topic information derived from acoustic and visual modalities is more discriminative and robust for emotion detection compared to language-based topics. This is attributed to the more direct and consistent nature of emotional cues in these modalities (e.g., smiles).
  • Model-Agnosticism is a Strength: TopicDiff's ability to be easily integrated with existing MCE architectures without requiring significant modifications is a major advantage.

Key Results

  • TopicDiff consistently improves the performance of various baseline MCE approaches across all tested datasets.
  • On the topic-density M3ED* dataset, TopicDiff applied to MMGCN achieved a 3.21% improvement in Weighted-F1 (W-F1) score over the baseline (p-value < 0.01). Similar improvements were observed on MELD and IEMOCAP.
  • TopicDiff showed significant gains in detecting challenging emotions, with up to a 9.55% improvement for the "fear" emotion when applied to MMGCN.
  • Ablation studies confirmed that the diffusion model component (TDB) is essential for TopicDiff's performance gains, mitigating diversity deficiency.
  • Qualitative analysis showed that combining multimodal topic clues (language, acoustic, vision) significantly boosted the prediction of the "joy" emotion, outperforming models that relied solely on language topics.

Limitations and Future Work

  • Limitations: The paper acknowledges that while TopicDiff improves performance, the results of reproduced baselines were sometimes slightly lower than reported in original papers due to differing experimental environments.
  • Future Work: The authors suggest several promising directions:
  • Incorporating additional information like speaker personality and posture.
  • Leveraging large-scale multimodal video pre-training to enhance modality representations.
  • Extending TopicDiff to other multimodal conversational tasks, such as psychological counseling and depression detection.

Summary updated

BibTeX
@inproceedings{luo2024topicdiff,
  author = {Jiamin Luo and Jingjing Wang and Guodong Zhou},
  title = {{TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure