Toward a Controllable Disentanglement Network

Zengjie Song, Oluwasanmi Koyejo, Jiangshe ZhangIEEE Trans. Cybernetics 2022 · OpenReview

TL;DR

This paper introduces a Controllable Disentanglement Network (CDNet) that tackles two key challenges in learning disentangled image representations: controlling the degree of disentanglement during image editing and balancing disentanglement with image reconstruction quality. CDNet uses a novel distance covariance-based regularization and a soft target representation to achieve controllable attribute manipulation and improved image generation.

What Problem Does This Paper Address?

Learning disentangled representations is crucial for making machine learning models more interpretable and controllable. Ideally, a disentangled representation would allow us to manipulate specific attributes of an image (like changing a smile on a face) without affecting other aspects (like the person's identity or background). However, existing methods struggle with two main issues:

  1. Controlling the intensity of disentanglement: Most models can change an attribute, but they can't precisely control how much it changes. For example, they might turn a neutral expression into a smile, but not generate a range of smiles from subtle to broad.
  2. Balancing disentanglement and image quality: Methods that achieve good disentanglement often produce blurry or distorted images. Conversely, models that generate high-quality images might not be well-disentangled.

This paper aims to bridge this gap by developing a model that offers fine-grained control over attribute manipulation while maintaining high image fidelity.

Methods

CDNet combines an Autoencoder (AE) with a Generative Adversarial Network (GAN) in a novel architecture. Here's a breakdown of its key components and novelties:

  • Dual Encoders:
    • Ency: Learns a "soft target representation" ($\hat{y}$) that captures class or attribute information. This is trained using a classification loss.
  • Encz: Learns a latent representation ($z$) that is independent of $\hat{y}$.
  • Novel Decorrelation Regularization: The core of disentanglement is achieved through a distance covariance (dCov) based decorrelation loss ($L_{dcorr}$). Unlike simpler correlation measures, dCov aims to enforce statistical independence between the two representations ($\hat{y}$ and $z$), which is crucial for better disentanglement.
  • Soft Target Representation for Control: Instead of using discrete labels, CDNet uses a continuous, real-valued soft target representation ($\hat{y}$). This allows for fine-grained control during image editing by directly manipulating the values in $\hat{y}$ to adjust the intensity of specific attributes.
  • AE-GAN Integration: The AE decoder (Dec) and the GAN generator (Gen) are merged into a single component (Dec/Gen) with shared parameters. This joint training helps improve the perceptual quality of generated images by leveraging the GAN's ability to produce realistic outputs while the AE ensures reconstruction.
  • Dual Reconstruction Losses: CDNet uses both a pixel-wise reconstruction loss ($L_{pix}^{rec}$) and a feature-matching reconstruction loss ($L_{feat}^{rec}$) derived from the GAN discriminator. This combination helps preserve both global identity and local details, leading to higher quality reconstructions.
  • Classification-based Evaluation Protocol: A novel protocol is introduced to quantitatively measure disentanglement strength. It involves training a classifier to detect specific attributes in synthesized images, with lower error rates indicating better disentanglement.

Datasets

The paper uses two standard datasets for evaluation:

  • MNIST: For handwritten digit generation, demonstrating disentanglement of handwriting styles.
  • CelebA: For face image manipulation, showcasing disentanglement of facial attributes like "blond hair," "eyeglasses," and "smiling."

Key Insights and Ideas

  • Distance Covariance for True Independence: The paper highlights that distance covariance is a more powerful tool than simple cross-covariance for enforcing statistical independence between latent representations, leading to better disentanglement.
  • Soft Targets Enable Fine-Grained Control: Moving beyond discrete labels to continuous soft target representations is key to controlling the intensity of manipulated attributes, enabling smoother transitions and more nuanced editing.
  • AE-GAN Synergy for Quality: Combining AE and GAN architectures, specifically by sharing parameters between the decoder and generator, effectively leverages the strengths of both: AE for reconstruction and GAN for perceptual realism.
  • Dual Reconstruction Losses are Crucial: Using both pixel-wise and feature-wise reconstruction losses is essential for preserving both the overall identity and fine details of the original image.
  • Classification as a Disentanglement Metric: The proposed classification-based evaluation protocol offers a quantitative way to assess disentanglement, moving beyond purely qualitative visual inspection.

Key Results

  • Improved Reconstruction Quality: CDNet significantly outperforms baseline models (AE-XCov, IcGAN, VAE/GAN) on standard image quality metrics (RMSE, PSNR, SSIM) for both MNIST and CelebA datasets.
  • Effective Disentanglement: CDNet successfully generates digits with different handwriting styles and synthesizes face images with desired attributes and intensities, preserving core identity better than baselines.
  • Controllable Attribute Intensity: Experiments show that CDNet can generate a range of attribute intensities, from weak to strong, by adjusting the soft target representation.
  • Quantitative Disentanglement Evaluation: The proposed classification protocol demonstrates that CDNet-dCov generally achieves lower classification error rates, indicating superior disentanglement compared to CDNet-XCov, especially with stronger regularization.
  • Ablation Studies: The study confirms the importance of each component: pixel-wise reconstruction for identity, feature-wise reconstruction for details, and decorrelation regularization for disentanglement.

Limitations and Future Work

  • Supervised Learning Dependency: The current model relies on labeled data for training the Ency encoder. The authors acknowledge that acquiring large labeled datasets is costly and suggest exploring semi-/weakly-supervised learning approaches for future work.
  • Extending to Other Domains: While demonstrated on digits and faces, extending the model to other data modalities (e.g., text, audio) or more complex image domains could be a future direction.
  • Further Exploration of GAN Architectures: The paper mentions that other GAN architectures could be used for improved perceptual quality, suggesting room for further optimization.
  • Investigating Inductive Biases: The paper briefly touches upon the theoretical impossibility of unsupervised disentanglement without inductive biases, hinting at future research into how these biases can be effectively incorporated.

Summary updated

Abstract

This article addresses two crucial problems of learning disentangled image representations, namely, controlling the degree of disentanglement during image editing, and balancing the disentanglement strength and the reconstruction quality. To encourage disentanglement, we devise distance covariance-based decorrelation regularization. Further, for the reconstruction step, our model leverages a soft target representation combined with the latent image code. By exploring the real-valued space of the soft target representation, we are able to synthesize novel images with the designated properties. To improve the perceptual quality of images generated by autoencoder (AE)-based models, we extend the encoder–decoder architecture with the generative adversarial network (GAN) by collapsing the AE decoder and the GAN generator into one. We also design a classification-based protocol to quantitatively evaluate the disentanglement strength of our model. The experimental results showcase the benefits of the proposed model.
BibTeX
@article{song2022toward,
  author = {Zengjie Song and Oluwasanmi Koyejo and Jiangshe Zhang},
  title = {{Toward a Controllable Disentanglement Network}},
  journal = {IEEE Trans. Cybernetics},
  year = {2022},
  url = {https://openreview.net/forum?id=UHxn-jIgFc},
}
Figure