Abstract
Multi-view clustering (MVC) aims to group data points by identifying commonalities across multiple observations. However, performance is often hindered by the entanglement of cluster-relevant information with view-specific "peculiar" details. We present Multi-VAE, a generative framework that explicitly disentangles these factors into discrete view-common and continuous view-peculiar latent variables. By employing a controlled information capacity mechanism, Multi-VAE achieves superior clustering accuracy and provides explainable visual representations of the learned latent space.
Introduction
In real-world scenarios, objects are often observed through multiple modalities or viewpoints. MVC seeks to leverage this redundancy to improve clustering. A primary challenge is that each view contains unique information—such as camera angles, stroke thickness, or lighting—that is irrelevant to the object's identity. When these "peculiar" features are fused with "common" features, they act as noise, degrading clustering performance. Multi-VAE addresses this by separating the "what" (the cluster identity) from the "how" (the view-specific style).
Methodology
Multi-VAE utilizes a Variational Autoencoder (VAE) architecture with multiple encoder-decoder pairs, one for each view. The core innovation lies in the design of the latent space, which is split into two distinct types of variables:
- View-Common Variable ($c$): This variable is inferred from the embeddings of all views. It uses a Gumbel-Softmax distribution, a differentiable method to model discrete categories, designed to capture the shared cluster identity.
- View-Peculiar Variables ($z^v$): Each view has its own $z^v$, inferred only from that specific view. These follow a continuous Gaussian distribution and are intended to capture style, pose, or lighting.
To ensure effective separation, the model uses a controlled information capacity constraint on the KL divergence terms. By gradually increasing this capacity during training, the model is forced to prioritize the most important information, effectively pushing discrete cluster data into $c$ and residual continuous noise into $z^v$.
Results and Discussion
The model was evaluated on several benchmarks, including Multi-MNIST, Multi-Fashion, and the newly constructed Digit-Product dataset. Multi-VAE consistently outperformed state-of-the-art methods. For instance, on the challenging Digit-Product dataset, Multi-VAE achieved 92.5% accuracy, significantly higher than the 79.8% achieved by the next best deep method (DEMVC).
A key advantage of Multi-VAE is its explainability through synthesis. Because it is a generative model, the learned representations can be visualized by fixing the cluster variable $c$ and traversing the continuous latent space $z^v$. This process demonstrates that the model successfully disentangles visual factors such as orientation, thickness, and scale from the object's identity.
Conclusion
Multi-VAE provides a robust and interpretable solution for multi-view clustering by leveraging the natural distinction between discrete cluster identities and continuous visual variations. While the model currently requires a predefined number of clusters ($K$), its linear complexity relative to the number of data points makes it highly scalable. Future work will explore automated cluster estimation and methods to handle incomplete multi-view data.
Summary updated