Introduction
Disentanglement in computer vision refers to the ability to decompose an image into independent factors of variation, such as shape, color, and pose. While Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have made significant progress, they often struggle with optimization complexity, resource inefficiency, and a failure to capture global dataset structures.
To address these challenges, we present ContraLORD, a framework that integrates latent optimization with dual-level contrastive learning. ContraLORD aims to create a unified, efficient encoder-decoder structure capable of dynamically learning features across large-scale datasets. By applying contrastive losses to both content and residual features, the model achieves state-of-the-art performance in factor separation and linear classification tasks.
Methodology
The ContraLORD framework operates in two distinct stages: Embedding Optimization and Encoder Pre-training.
1. Embedding Optimization
In the first stage, rather than training an encoder directly, the model utilizes a generator to identify the optimal discriminative and generative embeddings for each image through latent optimization. The generative embedding is partitioned into two components: Content (representing class or identity) and Residual (capturing uncorrelated information such as background or lighting).
2. Encoder Pre-training
The second stage involves training a generalized encoder to predict these optimized embeddings. The core innovation is the introduction of dual-level contrastive learning:
- Content-level Contrastive Loss: Employs a momentum encoder and a negative sample queue to align similar content features while distancing dissimilar ones.
- Residual-level Contrastive Loss: Applies a similar contrastive logic to residual features, ensuring the "leftover" information is structured and disentangled.
By utilizing momentum encoders (inspired by MoCo), ContraLORD can compare queries against thousands of negative samples, facilitating the learning of a more uniform and structured latent space.
Experimental Setup
The model was evaluated across ten diverse benchmarks to assess both classification and disentanglement capabilities:
- Linear Classification: CIFAR-10, CIFAR-100, and ImageNet-100.
- Synthetic Disentanglement: Shapes3D, Cars3D, dSprites, and SmallNorb.
- Real-world Image Datasets: CelebA, AFHQ (animal faces), and CelebA-HQ.
Results and Key Insights
ContraLORD demonstrates significant improvements over existing methods in both representation quality and downstream task performance.
Quantitative Performance
As shown in the table below, ContraLORD consistently outperforms baseline methods like LORD, MoCo, and SimCLR across multiple classification benchmarks.
Key results include:
- Classification Gains: ContraLORD surpassed the LORD baseline by 8.88% on CIFAR-10 and 9.35% on CIFAR-100 in linear top-1 accuracy.
- Disentanglement Excellence: On the Shapes3D dataset, the model achieved near-perfect scores (1.0) in DCI and MIG metrics, significantly outperforming previous state-of-the-art methods.
- High-Fidelity Synthesis: On the AFHQ dataset, ContraLORD achieved a superior FID score of 15.8 and a higher LPIPS of 0.53 compared to OverLORD, indicating better image quality and more meaningful feature separation.
Insights on Latent Space
A critical finding is that uniformity correlates with disentanglement. By using a Gaussian potential kernel to measure "smoothness," the authors found that ContraLORD produces significantly smoother latent spaces than its predecessors. Furthermore, the application of contrastive learning to residual factors—traditionally treated as noise—proved effective for organizing secondary attributes.
Discussion and Future Work
While ContraLORD sets a new benchmark, several areas for future exploration remain. The model's performance is currently sensitive to the weighting of contrastive losses, suggesting that dataset-specific tuning could yield further gains. Additionally, the two-stage training process, while effective, adds complexity; future research may focus on developing more integrated, single-stage training regimes. Finally, investigating the model's ability to discover entirely novel disentangled factors in unlabelled, "in-the-wild" data remains a promising direction.
Summary updated