Towards Generalized Multi-stage Clustering: Multi-view Self-distillation

Jiatai Wang, Zhiwei Xu, Xin Wang, Li, TaoarXiv 2023 · DOI · arXiv

Abstract

Existing multi-stage clustering methods independently learn salient features from multiple views and then perform the clustering task. Multi-view clustering (MVC) aims at exploring common semantics and pseudo-labels from multiple views in a self-supervised manner. However, limited by noisy data and inadequate feature learning, such paradigms often generate overconfident pseudo-labels that misguide the model, leading to inaccurate predictions. To alleviate this, we propose DistilMVC, a novel multi-stage deep MVC framework that introduces multi-view self-distillation to distill "dark knowledge" from label distributions. By exploring common semantics through hierarchical contrastive learning and maximizing mutual information between views, DistilMVC enhances robustness and predictive capabilities. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple real-world datasets.

The Challenge: Overconfident Pseudo-labels

In unsupervised Multi-view Clustering (MVC), models group data by integrating information from various sources (e.g., different image features or modalities). A common approach involves a multi-stage process: learning representations, generating "pseudo-labels" as cluster assignments, and refining the model based on these labels.

The primary bottleneck is overconfidence. High-dimensional data is frequently noisy; when a model becomes too certain about an incorrect pseudo-label, it enters a feedback loop of error. This bias accumulation degrades feature learning and final clustering accuracy. DistilMVC aims to break this cycle by introducing a more flexible, "soft" supervision signal.

Conceptual comparison: Pseudo-labels vs. Dark Knowledge distillation.

The DistilMVC Framework

To break the cycle of overconfident errors, DistilMVC replaces rigid, "hard" pseudo-labels with Self-Distillation.

1. Teacher-Student Architecture

The framework employs a student network that learns the clustering task and a teacher network updated via an Exponential Moving Average (EMA) of the student's weights. The teacher provides a stable, nuanced target for the student.

2. Distilling Dark Knowledge

Instead of a binary cluster assignment, the teacher produces a probability distribution—termed "dark knowledge." This distribution captures the underlying semantic relationships between clusters (e.g., noting that a "dog" image shares some features with "cat" clusters), which hard labels ignore. The student is trained to match this distribution using Kullback–Leibler (KL) Divergence, acting as a powerful regularizer against model collapse.

3. Hierarchical Contrastive Learning

DistilMVC ensures consistency across views at two distinct levels:

  • High-dimensional Subspace: A contrastive loss ensures that different views of the same instance are mapped close together in the feature space.
  • Low-dimensional Latent Space: Using Invariant Information Clustering (IIC), the model maximizes the statistical dependency (Mutual Information) between views to refine cluster boundaries.
The DistilMVC architecture, showing the flow from multi-view inputs to hierarchical contrastive learning and self-distillation.

Experimental Results

The model was evaluated on eight real-world datasets, including MNIST-USPS, BDGP, and various versions of Caltech-101.

  • State-of-the-Art Performance: DistilMVC consistently outperformed leading methods like MFLVC and COMPLETER. Notably, on the Caltech-4V dataset, it achieved a 7.6% improvement in accuracy over the next best method.
  • Scalability: Unlike many MVC methods that struggle with the "curse of dimensionality" as views increase, DistilMVC's performance remained stable and even improved when moving from 2-view to 5-view configurations.
  • Cluster Purity: A significant finding was the near-perfect one-to-one mapping between predicted clusters and ground-truth labels, evidenced by identical Accuracy (ACC) and Purity (PUR) scores on several benchmarks.

Conclusion

DistilMVC addresses the fundamental flaw of overconfidence in multi-stage clustering. By leveraging self-distillation and hierarchical consistency, it transforms noisy signals into robust "dark knowledge," setting a new standard for generalized multi-view clustering. Future work will explore extending this framework to handle incomplete datasets and cross-modal retrieval tasks.

Summary updated

Abstract

Existing multi-stage clustering methods independently learn the salient features from multiple views and then perform the clustering task. Particularly, multi-view clustering (MVC) has attracted a lot of attention in multi-view or multi-modal scenarios. MVC aims at exploring common semantics and pseudo-labels from multiple views and clustering in a self-supervised manner. However, limited by noisy data and inadequate feature learning, such a clustering paradigm generates overconfident pseudo-labels that mis-guide the model to produce inaccurate predictions. Therefore, it is desirable to have a method that can correct this pseudo-label mistraction in multi-stage clustering to avoid the bias accumulation. To alleviate the effect of overconfident pseudo-labels and improve the generalization ability of the model, this paper proposes a novel multi-stage deep MVC framework where multi-view self-distillation (DistilMVC) is introduced to distill dark knowledge of label distribution. Specifically, in the feature subspace at different hierarchies, we explore the common semantics of multiple views through contrastive learning and obtain pseudo-labels by maximizing the mutual information between views. Additionally, a teacher network is responsible for distilling pseudo-labels into dark knowledge, supervising the student network and improving its predictive capabilities to enhance the robustness. Extensive experiments on real-world multi-view datasets show that our method has better clustering performance than state-of-the-art methods.
BibTeX
@misc{wang2023towards,
  author = {Jiatai Wang and Zhiwei Xu and Xin Wang and Li, Tao},
  title = {{Towards Generalized Multi-stage Clustering: Multi-view Self-distillation}},
  year = {2023},
  eprint = {2310.18890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18890},
}
Figure