Introduction
Multimodal Intent Detection (MID) is a critical component of modern task-oriented dialogue systems, requiring the integration of text, video, and audio to accurately capture user intent. However, existing models often struggle with two primary issues: entanglement and causal ambiguity. Semantic information (the core intent) is frequently intertwined with modality-specific noise, such as background lighting or acoustic pitch. Furthermore, standard end-to-end training can lead to "shortcut learning," where models rely on spurious correlations rather than the true causal impact of specific features.
To address these challenges, we present DuoDN (Dual-oriented Disentangled Network), a framework that explicitly separates semantic and modality-specific features while employing counterfactual interventions to ensure robust causal reasoning.
Methodology
The DuoDN architecture consists of two innovative components designed to disentangle representations and enforce causal learning.
Dual-oriented Disentangled Encoder
This module employs specialized encoders to map multimodal inputs into two distinct subspaces:
- Semantics-oriented Subspace: Captures shared information across modalities (e.g., the concept of a "greeting" present in both text and gesture). Semantic-level contrastive learning is used to align these cross-modal meanings.
- Modality-oriented Subspace: Captures unique, non-redundant traits specific to a single channel, such as a particular facial expression or vocal tone.
Counterfactual Intervention Module (CIM)
To prevent the model from relying on shortcuts, the CIM introduces "causal pressure." During training, the module injects confounders—random noise sampled from a Gaussian distribution—to simulate counterfactual scenarios: "What would the intent be if the input were different?" By maximizing the discrepancy between original and counterfactual predictions, the model is forced to focus on the significant, disentangled features that truly drive intent.
Experiments and Results
DuoDN was evaluated on several tri-modal benchmarks, including MIntRec, MIntRec 2.0 (which includes out-of-scope samples), and MELD-DA.
Key Performance Metrics
- State-of-the-Art Results: DuoDN outperformed previous leaders like TCL-MAP on the MIntRec benchmark, achieving a 1.38% improvement in accuracy and a 1.16% increase in weighted F1-score.
- Out-of-Scope (OOS) Detection: On MIntRec 2.0, the model demonstrated superior performance in identifying unknown intents, reaching an F1-OOS of 63.95%.
- Ablation Study: Removing the CIM resulted in a performance drop of approximately 2% across all metrics, confirming the necessity of causal intervention for effective reasoning.
Discussion and Insights
The success of DuoDN highlights several key insights in multimodal learning. First, while text remains the dominant modality for intent detection, modality-specific features from audio and video act as "kingmakers" in ambiguous cases, such as distinguishing a joke from a complaint. Second, the use of causal inference through the CIM significantly improves model robustness and generalization, particularly for OOS detection.
Despite these gains, limitations remain. The model currently focuses on single-turn interactions and can struggle with complex social nuances like sarcasm or taunting. Future work will explore multi-turn context and data augmentation to address dataset biases.
Conclusion
DuoDN advances multimodal intent detection by combining explicit feature disentanglement with counterfactual reasoning. By moving beyond simple correlation and toward causal understanding, the framework achieves superior performance and robustness across diverse benchmarks.
Summary updated