Abstract
Existing multi-stage clustering methods independently learn salient features from multiple views and then perform the clustering task. Multi-view clustering (MVC) aims at exploring common semantics and pseudo-labels from multiple views in a self-supervised manner. However, limited by noisy data and inadequate feature learning, such paradigms often generate overconfident pseudo-labels that misguide the model, leading to inaccurate predictions. To alleviate this, we propose DistilMVC, a novel multi-stage deep MVC framework that introduces multi-view self-distillation to distill "dark knowledge" from label distributions. By exploring common semantics through hierarchical contrastive learning and maximizing mutual information between views, DistilMVC enhances robustness and predictive capabilities. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple real-world datasets.
The Challenge: Overconfident Pseudo-labels
In unsupervised Multi-view Clustering (MVC), models group data by integrating information from various sources (e.g., different image features or modalities). A common approach involves a multi-stage process: learning representations, generating "pseudo-labels" as cluster assignments, and refining the model based on these labels.
The primary bottleneck is overconfidence. High-dimensional data is frequently noisy; when a model becomes too certain about an incorrect pseudo-label, it enters a feedback loop of error. This bias accumulation degrades feature learning and final clustering accuracy. DistilMVC aims to break this cycle by introducing a more flexible, "soft" supervision signal.
The DistilMVC Framework
To break the cycle of overconfident errors, DistilMVC replaces rigid, "hard" pseudo-labels with Self-Distillation.
1. Teacher-Student Architecture
The framework employs a student network that learns the clustering task and a teacher network updated via an Exponential Moving Average (EMA) of the student's weights. The teacher provides a stable, nuanced target for the student.
2. Distilling Dark Knowledge
Instead of a binary cluster assignment, the teacher produces a probability distribution—termed "dark knowledge." This distribution captures the underlying semantic relationships between clusters (e.g., noting that a "dog" image shares some features with "cat" clusters), which hard labels ignore. The student is trained to match this distribution using Kullback–Leibler (KL) Divergence, acting as a powerful regularizer against model collapse.
3. Hierarchical Contrastive Learning
DistilMVC ensures consistency across views at two distinct levels:
- High-dimensional Subspace: A contrastive loss ensures that different views of the same instance are mapped close together in the feature space.
- Low-dimensional Latent Space: Using Invariant Information Clustering (IIC), the model maximizes the statistical dependency (Mutual Information) between views to refine cluster boundaries.
Experimental Results
The model was evaluated on eight real-world datasets, including MNIST-USPS, BDGP, and various versions of Caltech-101.
- State-of-the-Art Performance: DistilMVC consistently outperformed leading methods like MFLVC and COMPLETER. Notably, on the Caltech-4V dataset, it achieved a 7.6% improvement in accuracy over the next best method.
- Scalability: Unlike many MVC methods that struggle with the "curse of dimensionality" as views increase, DistilMVC's performance remained stable and even improved when moving from 2-view to 5-view configurations.
- Cluster Purity: A significant finding was the near-perfect one-to-one mapping between predicted clusters and ground-truth labels, evidenced by identical Accuracy (ACC) and Purity (PUR) scores on several benchmarks.
Conclusion
DistilMVC addresses the fundamental flaw of overconfidence in multi-stage clustering. By leveraging self-distillation and hierarchical consistency, it transforms noisy signals into robust "dark knowledge," setting a new standard for generalized multi-view clustering. Future work will explore extending this framework to handle incomplete datasets and cross-modal retrieval tasks.
Summary updated