Robust Contrastive Multi-view Kernel Clustering

Su Peng, Yixi Liu, Shujian Li, Shudong Huang, Jiancheng Lv, Wenjie Du, Yang Wang

Abstract

Multi-view Kernel Clustering (MKC) aims to integrate information from diverse data sources into a unified representation for improved grouping. While contrastive learning has emerged as a powerful paradigm for MKC, it is susceptible to the "false negative" problem, where semantically similar samples from the same cluster are erroneously pushed apart in the latent space. This paper presents Robust Contrastive Multi-view Kernel Learning (R-CMK), which addresses this issue through a novel "hole area" strategy. By identifying potential false negative pairs based on their proximity and reversing the optimization gradient, R-CMK preserves the intrinsic local structure of the data. Extensive experiments on seven benchmark datasets demonstrate that R-CMK consistently outperforms state-of-the-art methods in terms of accuracy, purity, and stability.

Introduction

In the field of machine learning, multi-view data refers to information collected from different perspectives—for example, a news story represented by its text, its images, and its citations. Multi-view Kernel Clustering (MKC) attempts to combine these views into a single high-dimensional Hilbert space to identify natural groupings.

A popular recent technique for this integration is Contrastive Learning, which pulls together positive pairs (different views of the same object) and pushes apart negative pairs (views of different objects). However, a major flaw exists: standard contrastive learning often treats different objects belonging to the same cluster as "negatives," pushing them apart and degrading clustering performance. These are known as False Negative Pairs (FNPs). R-CMK is designed to be a noise-robust version of contrastive learning that specifically addresses this trap.

Methodology: The "Hole Area" Strategy

R-CMK introduces a unified latent space where all views, regardless of their initial dimensions, can be compared directly. The core innovation of R-CMK is its treatment of negative pairs through a "hole area" strategy. Instead of applying a uniform repulsive force to all negative pairs, the algorithm analyzes their distance in the latent space.

If two "negative" samples are found to be very close together, the algorithm identifies them as potential False Negatives. For these suspicious pairs, R-CMK employs gradient reversal: instead of pushing them further apart, it stops the amplification of their distance or even nudges them closer. This prevents the model from destroying the natural clusters it is trying to find.

The authors compare the "vanilla" loss surface (left) with their robust version (right). Notice the "hole" in the center of (b) where the gradient direction changes to protect potential false negatives.

Experimental Evaluation

The researchers tested R-CMK across seven diverse benchmarks, including news stories (3 Sources), scientific papers (Cora), and face images (Yale). The performance gains were substantial. On the Cora dataset, R-CMK achieved 64.88% accuracy, compared to 59.34% for standard contrastive learning (CMK) and only 38.29% for traditional kernels.

Visual evidence further supports these results. Using t-SNE visualizations to project high-dimensional data into 2D, R-CMK demonstrates much tighter and more distinct clusters compared to both traditional methods and standard contrastive learning, particularly on complex datasets like Handwritten digits.

A t-SNE visualization on the Handwritten dataset. R-CMK (c) shows much clearer separation between digit classes than the traditional (a) or standard contrastive (b) approaches.

Discussion and Conclusion

R-CMK demonstrates that preserving the natural structure of data by allowing for "uncertainty" in close-range negative pairs is critical for effective clustering. By building high-quality kernels that are robust to noise, even simple clustering algorithms can outperform highly complex modern state-of-the-art methods. While the method faces challenges regarding computational complexity ($O(N^2)$) and sensitivity to projection dimensions, batching techniques and careful parameter tuning offer paths to scalability. Future work will focus on refining the identification of negative pairs and optimizing distance-based sample handling.

Summary updated

BibTeX
@inproceedings{peng2024robust,
  author = {Su Peng and Yixi Liu and Shujian Li and Shudong Huang and Jiancheng Lv and Wenjie Du and Yang Wang},
  title = {{Robust Contrastive Multi-view Kernel Clustering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=f2QMmOeyPW},
}
Figure