Introduction
In the field of Variational Autoencoders (VAEs), achieving disentanglement—the ability to separate independent factors of variation in data (e.g., shape vs. color)—remains a primary objective. While current state-of-the-art methods typically penalize Total Correlation (TC) to enforce latent independence, they often rely on the assumption that hidden features are either perfectly independent or identically distributed. In practice, feature dependencies are heterogeneous; some factors are strongly "coupled," leading to unstable training or suboptimal trade-offs between reconstruction quality and disentanglement.
The Contrastive Copula VAE (C2VAE) addresses this by explicitly modeling feature dependencies using Neural Gaussian Copulas. By distinguishing between "clean" factorized representations and "messy" coupled ones through a self-supervised contrastive classifier, the model effectively filters out entangled factors.
Methodology
The C2VAE architecture extends the standard VAE framework with two primary components designed to handle feature coupling:
- Neural Gaussian Copula: Copulas describe the relationship between random variables independently of their marginal distributions. C2VAE employs an auxiliary encoder branch to learn a "neural copula" that captures the correlation structure (the "glue") between hidden features, generating "coupled representations."
- Self-supervised Contrastive Classifier: A classifier ($\psi$) is trained to differentiate between standard factorized representations ($z_q$) and the coupled representations ($z_p$) generated by the copula.
- Contrastive Loss: By minimizing a contrastive loss ($\mathcal{L}_C$), the VAE is incentivized to "filter out" dependencies identified by the copula, forcing the primary latent space to retain only independent, explanatory features.
The training process involves a two-phase optimization: updating VAE parameters for reconstruction and independence (via TC loss), and updating the contrastive classifier to better distinguish clean from coupled factors.
Experimental Setup
C2VAE was evaluated on four standard disentanglement benchmarks:
- dSprites: 2D shapes (heart, oval, square).
- SmallNORB: 3D toy objects under varying conditions.
- 3D Shapes: Objects with varying floor/wall colors and scales.
- 3D Cars: 3D car models at different rotation angles.
Results and Discussion
C2VAE consistently outperformed baselines such as $\beta$-VAE, $\beta$-TCVAE, and FactorVAE. Key performance indicators included:
- Higher Disentanglement Scores: Superior SAP (Separated Attribute Predictability) and FAC (FactorVAE score) results. On dSprites, C2VAE achieved an SAP of 0.044, significantly higher than FactorVAE’s 0.028.
- Improved Factorization: The model achieved the lowest WCN (Normalized Wasserstein Distance), indicating a posterior distribution closer to perfect factorization.
- Training Stability: Unlike many TC-based VAEs that are sensitive to initialization, C2VAE demonstrates a smoother and more consistent learning curve.
Qualitatively, "latent traversal" visualizations showed that C2VAE is more effective at isolating difficult factors, such as "orientation" in dSprites, which frequently entangles with "shape" in competing models.
Conclusion
By moving beyond the standard IID assumption and explicitly modeling dependencies via Neural Gaussian Copulas, C2VAE provides a more robust framework for unsupervised disentanglement. The contrastive mechanism not only improves the quality of learned representations but also stabilizes the training process. Future work may explore more complex copula structures, such as Student-t or Mixture Copulas, for diverse data types like time-series or text.
Summary updated