Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clustering

Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, Lifang He

Abstract

Multi-view clustering (MVC) aims to group data points by identifying commonalities across multiple observations. However, performance is often hindered by the entanglement of cluster-relevant information with view-specific "peculiar" details. We present Multi-VAE, a generative framework that explicitly disentangles these factors into discrete view-common and continuous view-peculiar latent variables. By employing a controlled information capacity mechanism, Multi-VAE achieves superior clustering accuracy and provides explainable visual representations of the learned latent space.

Introduction

In real-world scenarios, objects are often observed through multiple modalities or viewpoints. MVC seeks to leverage this redundancy to improve clustering. A primary challenge is that each view contains unique information—such as camera angles, stroke thickness, or lighting—that is irrelevant to the object's identity. When these "peculiar" features are fused with "common" features, they act as noise, degrading clustering performance. Multi-VAE addresses this by separating the "what" (the cluster identity) from the "how" (the view-specific style).

Methodology

Multi-VAE utilizes a Variational Autoencoder (VAE) architecture with multiple encoder-decoder pairs, one for each view. The core innovation lies in the design of the latent space, which is split into two distinct types of variables:

  1. View-Common Variable ($c$): This variable is inferred from the embeddings of all views. It uses a Gumbel-Softmax distribution, a differentiable method to model discrete categories, designed to capture the shared cluster identity.
  2. View-Peculiar Variables ($z^v$): Each view has its own $z^v$, inferred only from that specific view. These follow a continuous Gaussian distribution and are intended to capture style, pose, or lighting.
Architecture of Multi-VAE showing the separation of view-common and view-peculiar variables through distinct latent paths.

To ensure effective separation, the model uses a controlled information capacity constraint on the KL divergence terms. By gradually increasing this capacity during training, the model is forced to prioritize the most important information, effectively pushing discrete cluster data into $c$ and residual continuous noise into $z^v$.

Results and Discussion

The model was evaluated on several benchmarks, including Multi-MNIST, Multi-Fashion, and the newly constructed Digit-Product dataset. Multi-VAE consistently outperformed state-of-the-art methods. For instance, on the challenging Digit-Product dataset, Multi-VAE achieved 92.5% accuracy, significantly higher than the 79.8% achieved by the next best deep method (DEMVC).

A key advantage of Multi-VAE is its explainability through synthesis. Because it is a generative model, the learned representations can be visualized by fixing the cluster variable $c$ and traversing the continuous latent space $z^v$. This process demonstrates that the model successfully disentangles visual factors such as orientation, thickness, and scale from the object's identity.

Latent traversals demonstrating the model's ability to disentangle specific visual factors like thickness, angle, and size while maintaining cluster identity.

Conclusion

Multi-VAE provides a robust and interpretable solution for multi-view clustering by leveraging the natural distinction between discrete cluster identities and continuous visual variations. While the model currently requires a predefined number of clusters ($K$), its linear complexity relative to the number of data points makes it highly scalable. Future work will explore automated cluster estimation and methods to handle incomplete multi-view data.

Summary updated

Abstract

Multi-view clustering, a long-standing and important research problem, focuses on mining complementary information from diverse views. However, existing works often fuse multiple views' representations or handle clustering in a common feature space, which may result in their entanglement especially for visual representations. To address this issue, we present a novel VAE-based multi-view clustering framework (Multi-VAE) by learning disentangled visual representations. Concretely, we define a view-common variable and multiple view-peculiar variables in the generative model. The prior of view-common variable obeys approximately discrete Gumbel Softmax distribution, which is introduced to extract the common cluster factor of multiple views. Meanwhile, the prior of view-peculiar variable follows continuous Gaussian distribution, which is used to represent each view's peculiar visual factors. By controlling the mutual information capacity to disentangle the view-common and view-peculiar representations, continuous visual information of multiple views can be separated so that their common discrete cluster information can be effectively mined. Experimental results demonstrate that Multi-VAE enjoys the disentangled and explainable visual representations, while obtaining superior clustering performance compared with state-of-the-art methods.
BibTeX
@inproceedings{xu2021multi,
  author = {Jie Xu and Yazhou Ren and Huayi Tang and Xiaorong Pu and Xiaofeng Zhu and Ming Zeng and Lifang He},
  title = {{Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clustering}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00910},
}
Figure