Learning Robust Representation for Clustering through Locality Preserving Variational Discriminative Network

Ruixuan Luo, Wei Li, Zhiyuan Zhang, Ruihan Bao, Keiko Harimoto, Xu SunarXiv 2020 · OpenReview

Abstract

Clustering is one of the fundamental problems in unsupervised learning. Recent deep learning based methods focus on learning clustering oriented representations. Among those methods, Variational Deep Embedding (VaDE) achieves great success in various clustering tasks by specifying a Gaussian Mixture prior to the latent space. However, VaDE suffers from two problems: 1) it is fragile to the input noise; 2) it ignores the locality information between the neighboring data points. In this paper, we propose a joint learning framework that improves VaDE with a robust embedding discriminator and a local structure constraint, which are both helpful to improve the robustness of our model. Experiment results on various vision and textual datasets demonstrate that our method outperforms the state-of-the-art baseline models in all metrics. Further detailed analysis shows that our proposed model is very robust to the adversarial inputs, which is a desirable property for practical applications.

Introduction

Clustering high-dimensional data like images or text is challenging due to the "curse of dimensionality," where standard distance metrics often fail. While deep learning methods like Variational Deep Embedding (VaDE) have improved performance by learning latent representations with a Gaussian Mixture Model (GMM) prior, they remain limited. Specifically, VaDE is highly sensitive to input noise because of its focus on exact reconstruction and often fails to capture local geometric structures between neighboring points, leading to inaccuracies at cluster boundaries.

Methodology

The proposed Locality Preserving Variational Discriminative Network (LPVDN) addresses these limitations through a joint optimization framework.

The LPVDN architecture integrates global structure modeling with a local structure modeling component and a robust discriminator.

The framework consists of three core components:

  1. Global Structure Modeling ($L_G$): Utilizes a Variational Autoencoder (VAE) with a GMM prior to capture the overall data distribution.
  2. Robust Embedding Discriminator ($L_{MI}$): Introduces a discriminator to maximize the Mutual Information (MI) between input data and latent representations. This approach prioritizes the underlying signal over noisy pixels, enhancing robustness against perturbations.
  3. Locality Preserving Module ($L_{LP}$): Inspired by t-SNE, this module uses a Student t-distribution to model pairwise distances in the latent space. By minimizing KL-divergence between high-dimensional and low-dimensional pairwise probabilities, it ensures local neighborhood relationships are preserved.

Experimental Results

LPVDN was evaluated on standard benchmarks including MNIST, Fashion-MNIST, and REUTERS datasets.

Clustering Performance

LPVDN consistently outperformed state-of-the-art baselines like VaDE and ClusterGAN. On MNIST, it achieved an accuracy (ACC) of 0.9716. On the more complex Fashion-MNIST, it reached 0.6837, significantly higher than VaDE's 0.6383. For textual data (REUTERS), LPVDN maintained high performance (ACC 0.8483), demonstrating robustness even with unbalanced datasets.

Robustness and Visualization

A key strength of LPVDN is its stability under noise. While baseline models' performance collapsed under Gaussian pixel corruption, LPVDN remained steady. Furthermore, the model excels at low-dimensional representation, maintaining high accuracy even when the latent dimension is reduced for visualization.

t-SNE visualization demonstrating the clear cluster separation achieved by LPVDN's latent representations.

As shown in the visualization, the combination of global priors and local constraints allows for clear cluster separation even in reduced dimensions, making it a powerful tool for data exploration.

Conclusion

LPVDN significantly advances deep clustering by integrating local structure preservation and mutual-information-based discrimination. While the pairwise distance modeling adds some computational overhead, the gains in accuracy and noise robustness are substantial. Future work may explore applying these robust representations to other unsupervised tasks such as anomaly detection.

Summary updated

Abstract

Clustering is one of the fundamental problems in unsupervised learning. Recent deep learning based methods focus on learning clustering oriented representations. Among those methods, Variational Deep Embedding achieves great success in various clustering tasks by specifying a Gaussian Mixture prior to the latent space. However, VaDE suffers from two problems: 1) it is fragile to the input noise; 2) it ignores the locality information between the neighboring data points. In this paper, we propose a joint learning framework that improves VaDE with a robust embedding discriminator and a local structure constraint, which are both helpful to improve the robustness of our model. Experiment results on various vision and textual datasets demonstrate that our method outperforms the state-of-the-art baseline models in all metrics. Further detailed analysis shows that our proposed model is very robust to the adversarial inputs, which is a desirable property for practical applications.
BibTeX
@misc{luo2020learning,
  author = {Ruixuan Luo and Wei Li and Zhiyuan Zhang and Ruihan Bao and Keiko Harimoto and Xu Sun},
  title = {{Learning Robust Representation for Clustering through Locality Preserving Variational Discriminative Network}},
  year = {2020},
  eprint = {2012.13489},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.13489},
}
Figure