Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

Hanlei Zhang, Hua Xu, Fei Long, Xin Wang, Kai Gao

Introduction

Discovering the underlying semantics—such as intents and dialogue acts—of human utterances without pre-labeled data is a fundamental challenge in natural language processing. Traditionally, this "semantics discovery" has been approached as a text-only clustering task. However, human communication is inherently multimodal; non-verbal cues like tone of voice or facial expressions often resolve ambiguities present in text alone. For instance, a phrase like "Oh, absolutely not" can shift from a literal refusal to a sarcastic joke based on the speaker's delivery.

Existing unsupervised methods often struggle with "blurry" cluster boundaries because they ignore these vital non-verbal signals. This paper introduces UMC, the first successful unsupervised multimodal clustering framework designed to leverage text, video, and audio to discover hidden semantic structures in human dialogue.

Multimodal instances help refine cluster boundaries compared to text-only clusters.

The UMC Framework

The proposed Unsupervised Multimodal Clustering (UMC) framework operates through a three-step iterative process designed to fuse modalities effectively and bootstrap learning from high-quality samples.

The complete UMC pipeline, including multimodal pre-training, sample selection, and iterative representation learning.

1. Multimodal Unsupervised Pre-training

To establish a robust initial feature space, UMC employs a novel non-verbal modality masking strategy. Recognizing text as the primary semantic anchor, the model creates "positive views" of an utterance by masking either the audio or video modality with zero vectors. Contrastive learning is then used to pull these different views of the same instance together, forcing the model to identify a shared semantic core across sensory inputs.

2. High-quality Sample Selection

In an unsupervised setting, data quality varies. UMC utilizes a curriculum-based approach to identify "high-quality" samples—those residing in high-density regions of their assigned clusters. Density is calculated based on distance to $K$-nearest neighbors. A key innovation is the automatic $K$ selection mechanism, which dynamically adjusts the neighborhood size for each cluster based on its internal cohesion, effectively handling imbalanced datasets.

3. Iterative Representation Learning

The model then engages in a "dual learning" strategy:

  • Supervised Contrastive Learning (SCL): Applied to high-quality samples using their cluster assignments as pseudo-labels to tighten cluster boundaries.
  • Unsupervised Contrastive Learning (UCL): Applied to lower-quality samples to maintain a well-distributed feature space and prevent representation collapse. This process repeats iteratively, gradually increasing the threshold $t$ to include more samples in the supervised phase until the entire dataset is processed.

Experimental Results

The authors evaluated UMC on three benchmark multimodal datasets: MIntRec (intent recognition), MELD-DA (dialogue acts), and IEMOCAP-DA (multi-party conversation).

UMC achieved significant performance gains, outperforming state-of-the-art methods like SCCL and USNID by 2–6% across standard clustering metrics (NMI, ARI, and Accuracy). Ablation studies comparing the full multimodal UMC against a text-only version showed a consistent 1–4% improvement, confirming that non-verbal modalities actively refine semantic boundaries rather than just providing redundant information. For example, in the "Prevent" intent category, non-verbal cues like nodding or physical blocking were essential for correct classification when the text was ambiguous.

Discussion and Limitations

The success of UMC highlights two critical insights: first, that text serves as a semantic base while audio and video act as essential refiners; and second, that geometric density can serve as a reliable proxy for ground-truth labels in unsupervised learning.

Despite these gains, challenges remain. Clustering performance on complex, real-world datasets like MELD-DA is still relatively low, reflecting the inherent difficulty of interpreting messy human dialogue. Additionally, the framework's reliance on specific hyper-parameters and temperature settings suggests a need for further research into more adaptive, parameter-free selection methods. Future work will explore handling extremely nuanced classes, such as sarcasm or humor, where semantic intent is particularly elusive.

Summary updated

BibTeX
@inproceedings{zhang2024unsupervised,
  author = {Hanlei Zhang and Hua Xu and Fei Long and Xin Wang and Kai Gao},
  title = {{Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.2},
}
Figure