Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

Hanlei Zhang, Hua Xu, Fei Long, Xin Wang, Kai Gao

Abstract

In human communication, semantics are often conveyed through a combination of verbal and non-verbal cues. Traditional intent discovery systems rely heavily on text, often failing to capture nuances like sarcasm or emotional subtext. This paper introduces Unsupervised Multimodal Clustering (UMC), a framework designed to discover semantic categories in multimodal utterances without human-provided labels. By integrating video, audio, and text through a three-stage process involving multimodal pre-training, dynamic sample selection, and sequential representation learning, UMC achieves significant performance improvements over existing state-of-the-art methods.

Introduction

The task of "intent discovery" involves grouping similar utterances into semantic categories. While supervised models have made strides in incorporating multimodal data, unsupervised scenarios—where no labels are available—remain challenging. Textual transcripts alone are often ambiguous; for instance, a "Joke" might be indistinguishable from a "Statement" without visual or auditory context. Existing unsupervised methods struggle with effective modality fusion and the presence of "noisy" data that can degrade the clustering process. UMC addresses these gaps by leveraging non-verbal cues as "tie-breakers" and implementing a robust sample selection mechanism.

The UMC Framework

The UMC algorithm operates in three distinct stages to refine multimodal representations and discover underlying intents.

The three-stage UMC framework: Multimodal Pre-training, Clustering/Selection, and Representation Learning.
  1. Multimodal Unsupervised Pre-training: The model begins by learning a joint representation space. It employs a "modality masking" technique where augmented views of a multimodal input (Text + Video + Audio) are created by masking either the video or audio components. Contrastive learning is then used to ensure that different views of the same utterance are mapped to similar representations, establishing a strong foundation for subsequent clustering.
  2. Dynamic High-Quality Sample Selection: To mitigate the impact of noisy data, UMC identifies "high-quality" samples—those located in the dense centers of clusters. Unlike static methods, UMC utilizes a density-based strategy that automatically adjusts the number of neighbors ($K$) for each cluster, accounting for the varying shapes and sizes of real-world semantic groups.

High-Quality Sample Selection

The density-based strategy for selecting high-quality samples using K-Nearest Neighbors and cohesion evaluation.
  1. Sequential Representation Learning: In the final stage, the model treats the cluster assignments of high-quality samples as "pseudo-labels." It applies a supervised contrastive loss to pull these similar items closer together. Conversely, for low-quality or ambiguous samples, an unsupervised contrastive loss is applied to push them apart, preventing them from blurring the boundaries between well-defined clusters. This curriculum-style approach allows the model to build a stable foundation before attempting to categorize more difficult data points.

Datasets and Evaluation

The framework was evaluated on three benchmark datasets: MIntRec (20 intent classes), MELD-DA (dialogue acts from Friends), and IEMOCAP-DA (scripted and improvised performances). Modern backbones, including the Swin Transformer for video and WavLM for audio, were utilized for feature extraction, providing a significant upgrade over older baseline methods.

Key Results and Insights

UMC consistently outperformed previous state-of-the-art methods, such as USNID, across four standard metrics: Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Accuracy (ACC), and F-measure (FMI).

  • Performance Gain: UMC demonstrated a 2-6% improvement in clustering scores.
  • Ablation Success: Studies revealed that the multimodal pre-training stage is critical; its removal led to a performance drop of 11-15%.
  • Visualization: t-SNE plots confirmed that UMC produces more distinct and cohesive data clusters compared to text-only or simpler multimodal approaches, creating tighter "islands" of data rather than overlapping clouds.

Conclusion

UMC represents a significant advancement in unsupervised intent discovery by effectively integrating non-verbal cues and managing data noise through dynamic sample selection. While challenges remain in categorizing highly complex intents like "Taunt" or "Oppose," the framework provides a robust path forward for machines to understand the multifaceted nature of human communication in real-world, unlabeled environments.

Summary updated

Abstract

Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions. Existing methods manifest limitations in leveraging nonverbal information for discerning complex semantics in unsupervised scenarios. This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training to establish well-initialized representations for subsequent clustering. An innovative strategy is proposed to dynamically select high-quality samples as guidance for representation learning, gauged by the density of each sample’s nearest neighbors. Besides, it is equipped to automatically determine the optimal value for the top-K parameter in each cluster to refine sample selection. Finally, both high- and low-quality samples are used to learn representations conducive to effective clustering. We build baselines on benchmark multimodal intent and dialogue act datasets. UMC shows remarkable improvements of 2-6% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain. The complete code and data are available at https://github.com/thuiar/UMC.
BibTeX
@inproceedings{zhang2024unsupervised,
  author = {Hanlei Zhang and Hua Xu and Fei Long and Xin Wang and Kai Gao},
  title = {{Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.2},
}
Figure