Representation Decomposition for Image Manipulation and Beyond

Shang‐Fu Chen, Jia-Wei Yan, Ya-Fan Su, Yu-Chiang Frank Wang

Abstract

Representation disentanglement aims at learning interpretable features, so that the output can be recovered or manipulated accordingly. While existing works like infoGAN and AC-GAN exist, they choose to derive disjoint attribute code for feature disentanglement, which is not applicable for existing/trained generative models. In this paper, we propose a decomposition-GAN (dec-GAN), which is able to achieve the decomposition of an existing latent representation into content and attribute features. Guided by the classifier pre-trained on the attributes of interest, our dec-GAN decomposes the attributes of interest from the latent representation, while data recovery and feature consistency objectives enforce the learning of our proposed method. Our experiments on multiple image datasets confirm the effectiveness and robustness of our dec-GAN over recent representation disentanglement models.

Introduction

In the field of Generative Adversarial Networks (GANs), representation disentanglement is a critical goal. It involves separating an image's fundamental identity (content) from its specific characteristics (attributes), such as a person's pose or expression. Standard approaches like infoGAN or AC-GAN achieve this by defining specific "attribute codes" during the initial training phase. However, this creates two major bottlenecks: inflexibility and high computational cost. If a new attribute needs to be manipulated, the entire model often requires retraining. Furthermore, training state-of-the-art GANs from scratch is resource-intensive.

The proposed dec-GAN (Decomposition-GAN) addresses these challenges by performing disentanglement on already-trained, "black-box" models. This allows for the "decomposition" of existing latent spaces without modifying the original generator's weights, making the process significantly faster and more flexible for downstream tasks.

Proposed Method: dec-GAN

The core innovation is the dec-GAN architecture, which consists of two encoders ($E_c$ for content and $E_a$ for attributes) and a pre-trained, fixed generator ($G$). Unlike previous methods that treat the latent code $z$ as a set of disjoint parts, dec-GAN assumes an additive relationship: $z = z_c + z_a$.

The dec-GAN architecture for decomposing latent representations into content ($z_c$) and attribute ($z_a$) features.

The workflow involves several key components:

  • Attribute Guidance: A pre-trained classifier ($C$) acts as a teacher, ensuring that images generated from the attribute code $z_a$ contain the intended characteristic (e.g., a specific pose or expression).
  • Data Recovery: A reconstruction loss ($\mathcal{L}_{rec}$) ensures that when $z_c$ and $z_a$ are combined and passed through the generator, the resulting image matches the original input $x$.
  • Feature Consistency: Consistency losses ($\mathcal{L}_c^{const}, \mathcal{L}_a^{const}$) prevent "information leakage" between the two codes. They ensure that swapping attribute codes between images preserves the respective content and attribute identities.
  • Distribution Matching: Using Kullback–Leibler divergence (KLD), the encoders map features to a standard normal distribution, facilitating smooth interpolation and manipulation.

Experiments and Results

The model was evaluated on two standard benchmarks: the MNIST dataset (70,000 handwritten digits) and the CMU Multi-PIE dataset (facial images focusing on pose and expression).

Quantitative Performance

On the CMU Multi-PIE dataset, features extracted by dec-GAN achieved exceptional accuracy: 99.74% for pose classification and 98.59% for identity classification. These results significantly outperform baselines like VAE-GAN and UFDN. Additionally, dec-GAN is reported to be 4 to 7 times faster to train than AC-GAN because the generative backbone remains fixed.

Qualitative Analysis

The effectiveness of the decomposition is visually confirmed through "attribute swapping" tasks, such as taking the identity of one person and applying the pose or expression of another.

Qualitative results of attribute swapping on the CMU Multi-PIE dataset, comparing dec-GAN variants against baselines.

As shown in the qualitative results, dec-GAN successfully separates identity from pose and expression, maintaining high visual fidelity compared to standard VAE-GAN approaches. This confirms that the content and attribute features were successfully disentangled within the fixed latent space.

Conclusion and Limitations

dec-GAN provides a robust framework for disentangling latent representations of pre-trained generative models. By utilizing an additive decomposition and a fixed generator, it offers a flexible and efficient alternative to training disentangled GANs from scratch. However, the quality of disentanglement is heavily reliant on the pre-trained attribute classifier. Future work may explore extending this framework to handle multiple attributes simultaneously or applying it to more complex, non-linear latent spaces.

Summary updated

Abstract

Representation disentanglement aims at learning interpretable features, so that the output can be recovered or manipulated accordingly. While existing works like infoGAN and AC-GAN exist, they choose to derive disjoint attribute code for feature disentanglement, which is not applicable for existing/trained generative models. In this paper, we propose a decomposition-GAN (dec-GAN), which is able to achieve the decomposition of an existing latent representation into content and attribute features. Guided by the classifier pre-trained on the attributes of interest, our dec-GAN decomposes the attributes of interest from the latent representation, while data recovery and feature consistency objectives enforce the learning of our proposed method. Our experiments on multiple image datasets confirm the effectiveness and robustness of our dec-GAN over recent representation disentanglement models.
BibTeX
@inproceedings{chen2021representation,
  author = {Shang‐Fu Chen and Jia-Wei Yan and Ya-Fan Su and Yu-Chiang Frank Wang},
  title = {{Representation Decomposition for Image Manipulation and Beyond}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506015},
}
Figure