Robust Contrastive Multi-view Kernel Clustering

Peng Su, Yixi Liu, Shujian Li, Shudong Huang, Jiancheng Lv

Abstract

Multi-view kernel clustering (MKC) aims to fully reveal the consistency and complementarity of multiple views in a potential Hilbert space, thereby enhancing clustering performance. The clustering results of most MKC methods are highly sensitive to the quality of the constructed kernels, as traditional methods independently compute kernel matrices for each view without fully considering complementary information across views. In previous contrastive multi-view kernel learning, the goal was to bring cross-view instances of the same sample closer during the kernel construction process while pushing apart instances across samples to achieve a comprehensive integration of cross-view information. However, its inherent drawback is the potential inappropriate amplification of distances between different instances of the same clusters (i.e., false negative pairs) during the training process, leading to a reduction in inter-class discriminability. To address this challenge, we propose a Robust Contrastive multi-view kernel Learning approach (R-CMK) against false negative pairs. It partitions negative pairs into different intervals based on distance or similarity, and for false negative pairs, reverses their optimization gradient. This effectively avoids further amplification of distances for false negative pairs while simultaneously pushing true negative pairs farther apart. We conducted comprehensive experiments on various MKC methods to validate the effectiveness of the proposed method.

Introduction

In machine learning, Multi-view Kernel Clustering (MKC) is used to group data originating from multiple sources, such as news stories represented by text, images, and citations. The goal is to find a "consensus" grouping that respects all these views. Recently, researchers have integrated Contrastive Learning to improve this process by pulling "positive pairs" (the same object in different views) closer and pushing "negative pairs" (different objects) farther apart.

However, a major challenge arises with False Negative Pairs (FNPs). In clustering, two different samples might actually belong to the same category (e.g., two different photos of the same dog breed). Standard contrastive learning treats them as negatives and forces them apart, which inadvertently destroys the cluster structure and reduces the "discriminability" of the data.

Methods: Robust Contrastive Learning

The researchers developed R-CMK, a "plug-and-play" kernel construction framework designed to be resilient against false negatives. The method involves three key steps:

  1. Latent Space Projection: Data from different views are projected into a unified latent space and normalized.
  2. The Robust Loss Function: Instead of a standard contrastive loss, R-CMK uses a specialized loss for negative pairs.
  3. The "Hole Area": If a negative pair is very similar (likely a false negative), the algorithm reverses the gradient. Instead of pushing them apart, it stops the separation or pulls them slightly closer.
  4. The "Deceleration Area": For pairs that are somewhat similar, it slows down the optimization to prevent aggressive separation.
  5. Kernel Construction: Once robust latent representations are learned, they are used to build high-quality kernel matrices for standard clustering algorithms.
The "Hole Area" concept in R-CMK's loss function (b) compared to traditional contrastive loss (a). The "hole" prevents the separation of highly similar false negative pairs.

Experimental Results

The authors evaluated R-CMK on seven diverse benchmarks, including news articles (3 Sources, BBCSport), citation networks (Cora), and image datasets (Handwritten, Yale, 100 Leaves).

Key Findings

  • Accuracy Boost: On the Cora dataset using the MKKM framework, R-CMK improved accuracy from 38.29% (traditional) to 64.88%.
  • Superiority over CMK: In almost every test case, R-CMK outperformed the standard contrastive method (CMK), proving that handling false negatives is essential.
  • Visual Proof: t-SNE visualizations of the Handwritten dataset show that R-CMK produces much tighter, more distinct clusters compared to traditional methods.
t-SNE visualizations comparing (a) Traditional, (b) CMK, and (c) R-CMK clustering results on the Handwritten dataset. R-CMK shows superior cluster separation.

Conclusion

The R-CMK approach demonstrates that reversing gradients for highly similar negative pairs preserves the natural structure of data. The research highlights that kernel quality is often more critical than the clustering algorithm itself. Future work will explore more sophisticated ways to identify negative pairs and refine robustness calculations to further reduce intra-cluster distances.

Summary updated

Abstract

Multi-view kernel clustering (MKC) aims to fully reveal the consistency and complementarity of multiple views in a potential Hilbert space, thereby enhancing clustering performance. The clustering results of most MKC methods are highly sensitive to the quality of the constructed kernels, as traditional methods independently compute kernel matrices for each view without fully considering complementary information across views. In previous contrastive multi-view kernel learning, the goal was to bring cross-view instances of the same sample closer during the kernel construction process while pushing apart instances across samples to achieve a comprehensive integration of cross-view information. However, its inherent drawback is the potential inappropriate amplification of distances between different instances of the same clusters (i.e., false negative pairs) during the training process, leading to a reduction in inter-class discriminability. To address this challenge, we propose a Robust Contrastive multi-view kernel Learning approach (R-CMK) against false negative pairs. It partitions negative pairs into different intervals based on distance or similarity, and for false negative pairs, reverses their optimization gradient. This effectively avoids further amplification of distances for false negative pairs while simultaneously pushing true negative pairs farther apart. We conducted comprehensive experiments on various MKC methods to validate the effectiveness of the proposed method. The code is available at https://github.com/Duo-laimi/rcmk_main.
BibTeX
@inproceedings{su2024robust,
  author = {Peng Su and Yixi Liu and Shujian Li and Shudong Huang and Jiancheng Lv},
  title = {{Robust Contrastive Multi-view Kernel Clustering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=f2QMmOeyPW},
}
Figure