Abstract
In human communication, semantics are often conveyed through a combination of verbal and non-verbal cues. Traditional intent discovery systems rely heavily on text, often failing to capture nuances like sarcasm or emotional subtext. This paper introduces Unsupervised Multimodal Clustering (UMC), a framework designed to discover semantic categories in multimodal utterances without human-provided labels. By integrating video, audio, and text through a three-stage process involving multimodal pre-training, dynamic sample selection, and sequential representation learning, UMC achieves significant performance improvements over existing state-of-the-art methods.
Introduction
The task of "intent discovery" involves grouping similar utterances into semantic categories. While supervised models have made strides in incorporating multimodal data, unsupervised scenarios—where no labels are available—remain challenging. Textual transcripts alone are often ambiguous; for instance, a "Joke" might be indistinguishable from a "Statement" without visual or auditory context. Existing unsupervised methods struggle with effective modality fusion and the presence of "noisy" data that can degrade the clustering process. UMC addresses these gaps by leveraging non-verbal cues as "tie-breakers" and implementing a robust sample selection mechanism.
The UMC Framework
The UMC algorithm operates in three distinct stages to refine multimodal representations and discover underlying intents.
- Multimodal Unsupervised Pre-training: The model begins by learning a joint representation space. It employs a "modality masking" technique where augmented views of a multimodal input (Text + Video + Audio) are created by masking either the video or audio components. Contrastive learning is then used to ensure that different views of the same utterance are mapped to similar representations, establishing a strong foundation for subsequent clustering.
- Dynamic High-Quality Sample Selection: To mitigate the impact of noisy data, UMC identifies "high-quality" samples—those located in the dense centers of clusters. Unlike static methods, UMC utilizes a density-based strategy that automatically adjusts the number of neighbors ($K$) for each cluster, accounting for the varying shapes and sizes of real-world semantic groups.
High-Quality Sample Selection
- Sequential Representation Learning: In the final stage, the model treats the cluster assignments of high-quality samples as "pseudo-labels." It applies a supervised contrastive loss to pull these similar items closer together. Conversely, for low-quality or ambiguous samples, an unsupervised contrastive loss is applied to push them apart, preventing them from blurring the boundaries between well-defined clusters. This curriculum-style approach allows the model to build a stable foundation before attempting to categorize more difficult data points.
Datasets and Evaluation
The framework was evaluated on three benchmark datasets: MIntRec (20 intent classes), MELD-DA (dialogue acts from Friends), and IEMOCAP-DA (scripted and improvised performances). Modern backbones, including the Swin Transformer for video and WavLM for audio, were utilized for feature extraction, providing a significant upgrade over older baseline methods.
Key Results and Insights
UMC consistently outperformed previous state-of-the-art methods, such as USNID, across four standard metrics: Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Accuracy (ACC), and F-measure (FMI).
- Performance Gain: UMC demonstrated a 2-6% improvement in clustering scores.
- Ablation Success: Studies revealed that the multimodal pre-training stage is critical; its removal led to a performance drop of 11-15%.
- Visualization: t-SNE plots confirmed that UMC produces more distinct and cohesive data clusters compared to text-only or simpler multimodal approaches, creating tighter "islands" of data rather than overlapping clouds.
Conclusion
UMC represents a significant advancement in unsupervised intent discovery by effectively integrating non-verbal cues and managing data noise through dynamic sample selection. While challenges remain in categorizing highly complex intents like "Taunt" or "Oppose," the framework provides a robust path forward for machines to understand the multifaceted nature of human communication in real-world, unlabeled environments.
Summary updated