Representation Disentanglement in Generative Models with Contrastive Learning

Shentong Mo, Zhun Sun, Chao Li

Introduction

Disentanglement in computer vision refers to the ability to decompose an image into independent factors of variation, such as shape, color, and pose. While Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have made significant progress, they often struggle with optimization complexity, resource inefficiency, and a failure to capture global dataset structures.

To address these challenges, we present ContraLORD, a framework that integrates latent optimization with dual-level contrastive learning. ContraLORD aims to create a unified, efficient encoder-decoder structure capable of dynamically learning features across large-scale datasets. By applying contrastive losses to both content and residual features, the model achieves state-of-the-art performance in factor separation and linear classification tasks.

Methodology

The ContraLORD framework operates in two distinct stages: Embedding Optimization and Encoder Pre-training.

1. Embedding Optimization

In the first stage, rather than training an encoder directly, the model utilizes a generator to identify the optimal discriminative and generative embeddings for each image through latent optimization. The generative embedding is partitioned into two components: Content (representing class or identity) and Residual (capturing uncorrelated information such as background or lighting).

2. Encoder Pre-training

The second stage involves training a generalized encoder to predict these optimized embeddings. The core innovation is the introduction of dual-level contrastive learning:

  • Content-level Contrastive Loss: Employs a momentum encoder and a negative sample queue to align similar content features while distancing dissimilar ones.
  • Residual-level Contrastive Loss: Applies a similar contrastive logic to residual features, ensuring the "leftover" information is structured and disentangled.
Overview of the ContraLORD framework, showing the two-stage process of embedding optimization and encoder pre-training.

By utilizing momentum encoders (inspired by MoCo), ContraLORD can compare queries against thousands of negative samples, facilitating the learning of a more uniform and structured latent space.

Experimental Setup

The model was evaluated across ten diverse benchmarks to assess both classification and disentanglement capabilities:

  • Linear Classification: CIFAR-10, CIFAR-100, and ImageNet-100.
  • Synthetic Disentanglement: Shapes3D, Cars3D, dSprites, and SmallNorb.
  • Real-world Image Datasets: CelebA, AFHQ (animal faces), and CelebA-HQ.

Results and Key Insights

ContraLORD demonstrates significant improvements over existing methods in both representation quality and downstream task performance.

Quantitative Performance

As shown in the table below, ContraLORD consistently outperforms baseline methods like LORD, MoCo, and SimCLR across multiple classification benchmarks.

Linear classification results on CIFAR and ImageNet-100 datasets.

Key results include:

  • Classification Gains: ContraLORD surpassed the LORD baseline by 8.88% on CIFAR-10 and 9.35% on CIFAR-100 in linear top-1 accuracy.
  • Disentanglement Excellence: On the Shapes3D dataset, the model achieved near-perfect scores (1.0) in DCI and MIG metrics, significantly outperforming previous state-of-the-art methods.
  • High-Fidelity Synthesis: On the AFHQ dataset, ContraLORD achieved a superior FID score of 15.8 and a higher LPIPS of 0.53 compared to OverLORD, indicating better image quality and more meaningful feature separation.

Insights on Latent Space

A critical finding is that uniformity correlates with disentanglement. By using a Gaussian potential kernel to measure "smoothness," the authors found that ContraLORD produces significantly smoother latent spaces than its predecessors. Furthermore, the application of contrastive learning to residual factors—traditionally treated as noise—proved effective for organizing secondary attributes.

Discussion and Future Work

While ContraLORD sets a new benchmark, several areas for future exploration remain. The model's performance is currently sensitive to the weighting of contrastive losses, suggesting that dataset-specific tuning could yield further gains. Additionally, the two-stage training process, while effective, adds complexity; future research may focus on developing more integrated, single-stage training regimes. Finally, investigating the model's ability to discover entirely novel disentangled factors in unlabelled, "in-the-wild" data remains a promising direction.

Summary updated

Abstract

Contrastive learning has shown its effectiveness in image classification and generation. Recent works apply contrastive learning to the discriminator of the Generative Adversarial Networks. However, there is little work exploring if contrastive learning can be applied to the encoderdecoder structure to learn disentangled representations. In this work, we propose a simple yet effective method via incorporating contrastive learning into latent optimization, where we name it ContraLORD. Specifically, we first use a generator to learn discriminative and disentangled embeddings via latent optimization. Then an encoder and two momentum encoders are applied to dynamically learn disentangled information across a large number of samples with content-level and residual-level contrastive loss. In the meanwhile, we tune the encoder with the learned embeddings in an amortized manner. We evaluate our approach on ten benchmarks regarding representation disentanglement and linear classification. Extensive experiments demonstrate the effectiveness of our ContraLORD on learning both discriminative and generative representations.
BibTeX
@inproceedings{mo2023representation,
  author = {Shentong Mo and Zhun Sun and Chao Li},
  title = {{Representation Disentanglement in Generative Models with Contrastive Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00158},
}
Figure