Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection

Zhanpeng Chen, Zhihong Zhu, Xianwei Zhuang, Zhiqi Huang, Yuexian ZouEMNLP 2024 · DOI

Introduction

Multimodal Intent Detection (MID) is a critical component of modern task-oriented dialogue systems, requiring the integration of text, video, and audio to accurately capture user intent. However, existing models often struggle with two primary issues: entanglement and causal ambiguity. Semantic information (the core intent) is frequently intertwined with modality-specific noise, such as background lighting or acoustic pitch. Furthermore, standard end-to-end training can lead to "shortcut learning," where models rely on spurious correlations rather than the true causal impact of specific features.

To address these challenges, we present DuoDN (Dual-oriented Disentangled Network), a framework that explicitly separates semantic and modality-specific features while employing counterfactual interventions to ensure robust causal reasoning.

Conceptual comparison between direct fusion and the proposed disentangled representation approach.

Methodology

The DuoDN architecture consists of two innovative components designed to disentangle representations and enforce causal learning.

Dual-oriented Disentangled Encoder

This module employs specialized encoders to map multimodal inputs into two distinct subspaces:

  1. Semantics-oriented Subspace: Captures shared information across modalities (e.g., the concept of a "greeting" present in both text and gesture). Semantic-level contrastive learning is used to align these cross-modal meanings.
  2. Modality-oriented Subspace: Captures unique, non-redundant traits specific to a single channel, such as a particular facial expression or vocal tone.

Counterfactual Intervention Module (CIM)

To prevent the model from relying on shortcuts, the CIM introduces "causal pressure." During training, the module injects confounders—random noise sampled from a Gaussian distribution—to simulate counterfactual scenarios: "What would the intent be if the input were different?" By maximizing the discrepancy between original and counterfactual predictions, the model is forced to focus on the significant, disentangled features that truly drive intent.

Overview of the DuoDN framework, including the disentangled encoder and counterfactual intervention module.

Experiments and Results

DuoDN was evaluated on several tri-modal benchmarks, including MIntRec, MIntRec 2.0 (which includes out-of-scope samples), and MELD-DA.

Key Performance Metrics

  • State-of-the-Art Results: DuoDN outperformed previous leaders like TCL-MAP on the MIntRec benchmark, achieving a 1.38% improvement in accuracy and a 1.16% increase in weighted F1-score.
  • Out-of-Scope (OOS) Detection: On MIntRec 2.0, the model demonstrated superior performance in identifying unknown intents, reaching an F1-OOS of 63.95%.
  • Ablation Study: Removing the CIM resulted in a performance drop of approximately 2% across all metrics, confirming the necessity of causal intervention for effective reasoning.

Discussion and Insights

The success of DuoDN highlights several key insights in multimodal learning. First, while text remains the dominant modality for intent detection, modality-specific features from audio and video act as "kingmakers" in ambiguous cases, such as distinguishing a joke from a complaint. Second, the use of causal inference through the CIM significantly improves model robustness and generalization, particularly for OOS detection.

Despite these gains, limitations remain. The model currently focuses on single-turn interactions and can struggle with complex social nuances like sarcasm or taunting. Future work will explore multi-turn context and data augmentation to address dataset biases.

Conclusion

DuoDN advances multimodal intent detection by combining explicit feature disentanglement with counterfactual reasoning. By moving beyond simple correlation and toward causal understanding, the framework achieves superior performance and robustness across diverse benchmarks.

Summary updated

BibTeX
@inproceedings{chen2024dual,
  author = {Zhanpeng Chen and Zhihong Zhu and Xianwei Zhuang and Zhiqi Huang and Yuexian Zou},
  title = {{Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.972},
}
Figure