DealMVC: Dual Contrastive Calibration for Multi-view Clustering

Xihong Yang, Jiaqi Jin, Siwei Wang, Ke Liang, Yue Liu, Yi Wen, Suyuan Liu, Sihang Zhou, Xinwang Liu, En Zhu

Introduction

Multi-view clustering (MVC) aims to partition data points into distinct groups by leveraging information from multiple sources or "views," such as different modalities of the same object (e.g., images and text) or different feature extraction techniques. While recent advancements in contrastive learning have significantly improved MVC performance, most existing methods focus primarily on instance-level consistency—ensuring that different views of the same sample are aligned in the latent space.

However, this focus often overlooks the broader semantic structure of the data. DealMVC (Dual Contrastive Calibration for Multi-view Clustering) addresses this limitation by introducing a calibration mechanism that ensures semantically similar samples, even those from different views, are correctly aligned. By anchoring feature similarities to high-confidence pseudo-labels at both local and global levels, DealMVC achieves state-of-the-art performance across diverse benchmarks.

The Problem: The Instance-Level Blind Spot

In traditional contrastive MVC, the objective is typically to "pull" the representations of the same sample from different views closer together. While effective for maintaining identity, this approach often neglects the relationship between different samples belonging to the same category. For example, if Sample A and Sample B both represent "cats" but appear in different views, standard contrastive methods may not explicitly encourage their representations to cluster together. This lack of cross-sample calibration limits the model's ability to capture the true underlying semantic structure of the dataset.

The DealMVC Framework

DealMVC addresses these challenges through a three-tiered architecture designed to extract, fuse, and calibrate multi-view data.

1. Feature Extraction and Latent Space Mapping

The model employs an AutoEncoder Module for each view. These encoders project raw features into a shared latent space, while decoders ensure that essential information is preserved through a reconstruction loss. This provides a stable foundation for subsequent fusion and calibration.

2. Adaptive Global Fusion

To create a unified representation of a data point, DealMVC utilizes an Adaptive Global Fusion Module. This module combines two distinct strategies:

  • Attention Mechanism: Assigns weights to different views based on their perceived importance.
  • Learnable View Sampling Network: Mitigates training biases by dynamically sampling views.
  • Regulatory Factor: A balancing parameter ensures the resulting fused feature ($Z^G$) is both robust and informative, preventing any single noisy view from dominating the representation.

3. Dual Contrastive Calibration (DCC)

The core innovation of DealMVC is the DCC mechanism, which operates at two levels:

  • Global Calibration: The model constructs a similarity graph of the fused features and aligns it with a "pseudo-label graph" built from high-confidence predictions. This forces the global feature space to respect cluster boundaries.
  • Local Calibration: This process is mirrored for individual views. By forcing view-specific features to align with the same reliable pseudo-labels, the model ensures that similar samples are pulled together even before the fusion stage.

Experimental Evaluation

The authors evaluated DealMVC on eight benchmark datasets, ranging from small-scale citation networks to large-scale image datasets:

  • Small/Medium Scale: BBCSport, Reuters, Caltech101_7, Cora, and Wiki.
  • Large Scale: Caltech101 (9,144 samples), Hdigit (10,000 samples), and STL10 (13,000 samples).

Key Results

DealMVC consistently outperformed nine state-of-the-art baselines, including MFLVC and CoMVC, across metrics such as Accuracy (ACC), Normalized Mutual Information (NMI), and Purity (PUR).

  • Superior Accuracy: On the STL10 dataset, DealMVC improved Accuracy by 5.30% and Purity by 5.70% over the next best competitor.
  • Near-Perfect Clustering: On the Hdigit dataset, the model achieved a remarkable 99.80% Accuracy.
  • Ablation Insights: Removing either the local or global calibration components resulted in a significant performance drop, confirming that the dual nature of the calibration is critical for capturing complex semantic structures.

Discussion and Future Work

DealMVC demonstrates that using high-confidence pseudo-labels as "soft" supervision signals can effectively bridge the gap between unsupervised feature learning and semantic clustering. By calibrating the neighborhood structure across views rather than just identifying individual samples, the model achieves a more nuanced understanding of the data.

Despite its success, the authors identify several areas for future research:

  • Threshold Optimization: The method relies on a threshold ($ au$) to select high-confidence pseudo-labels. Automating the selection of this threshold for new datasets remains a challenge.
  • Scalability: The construction of $N \times N$ similarity graphs is computationally intensive. Future iterations may explore anchor-based methods to handle datasets with hundreds of thousands of samples.
  • Incomplete Views: Currently, DealMVC assumes all views are present for all samples. Adapting the dual calibration framework to handle missing data views would broaden its applicability to real-world scenarios.

Summary updated

BibTeX
@inproceedings{yang2023dealmvc,
  author = {Xihong Yang and Jiaqi Jin and Siwei Wang and Ke Liang and Yue Liu and Yi Wen and Suyuan Liu and Sihang Zhou and Xinwang Liu and En Zhu},
  title = {{DealMVC: Dual Contrastive Calibration for Multi-view Clustering}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=ZVkFdJ1yCK},
}
Figure