Representation Decomposition For Image Manipulation And Beyond

Shang-Fu Chen, Jia-Wei Yan, Ya-Fan Su, Yu-Chiang Frank Wang

TL;DR

Decomposition-GAN (dec-GAN) is a novel framework that disentangles content and attribute features from the latent space of already trained generative models. By fixing the generator and using a pre-trained classifier to guide new encoders, it enables flexible image manipulation (like swapping poses or expressions) up to seven times faster than training traditional disentangled models from scratch.

What Problem Does This Paper Address?

Representation disentanglement is the "holy grail" of controllable image generation—it allows you to change a specific feature (like a person's smile) without affecting others (like their identity).

However, existing methods like infoGAN or AC-GAN have a major drawback: they are rigid. You must decide which attributes to disentangle before you start the computationally expensive process of training the GAN. If you decide later that you want to manipulate a different attribute, you have to retrain the entire model from square one. As generative models grow in scale, this "train-from-scratch" requirement becomes a massive bottleneck for researchers and engineers.

Methods

The authors propose dec-GAN, which performs feature decomposition on an existing latent representation $z$. Unlike typical methods that split a latent vector into disjoint halves, dec-GAN treats the latent code as an additive sum: $z = z_c + z_a$, where $z_c$ is the content code and $z_a$ is the attribute code.

The architecture, shown in Figure 1, consists of:

  • A Fixed Generator ($G$): An existing, pre-trained GAN (like VAE-GAN or Res-GAN).
  • Two Encoders ($E_c$ and $E_a$): These are trained to extract content and attribute features, respectively, from an input image.
  • A Pre-trained Classifier ($C$): This acts as a "teacher." It is pre-trained on the attribute of interest (e.g., a pose classifier) and provides a guidance loss ($L_{guide}$) to ensure $E_a$ captures the correct information.
Overview and architecture of dec-GAN

To keep the encoders "honest," the authors introduce feature consistency losses. These ensure that if you generate a new image using a specific $z_c$ and a random attribute, the content encoder $E_c$ should still be able to recover that exact $z_c$ from the resulting image.

Datasets

The model was evaluated on two standard benchmarks:

  1. MNIST: The classic handwritten digit dataset, used here to disentangle digit category (attribute) from stroke style/thickness (content).
  2. CMU Multi-PIE: A large-scale face dataset. The authors used a subset of 68,810 images to disentangle identity (content) from either pose or smiling expression (attributes).

Key Insights and Ideas

  • Additive Decomposition: Instead of forcing the latent space into pre-defined "slots" (disjoint codes), dec-GAN learns to decompose the existing, entangled latent vector $z$ into two additive components. This is more compatible with how many pre-trained GANs actually structure their latent spaces.
  • Post-hoc Flexibility: Because the generator is fixed, you can "plug in" different pre-trained classifiers to disentangle different attributes at will. You don't need to re-run the massive GAN training; you only train the lightweight encoders.
  • Efficiency through Initialization: By leveraging a pre-trained generator, the training process converges much faster. The authors report that dec-GAN is 4 to 7 times faster than AC-GAN.

Key Results

Qualitative Manipulation

The model successfully performs "attribute swapping." As seen in Figure 2 for MNIST, the model can take the "content" (style) of one digit and the "attribute" (identity) of another to create a hybrid.

Attribute swapping on MNIST

In Figure 3, the authors demonstrate this on faces using two different backbones (VAE-GAN and Res-GAN). They successfully swap poses and smiles while perfectly preserving the person's identity.

Pose and smile swapping on CMU Multi-PIE

Quantitative Performance

The authors tested how "pure" the disentangled features were by using them for classification tasks on the Multi-PIE dataset:

  • Pose Classification: The attribute code $z_a$ achieved 99.74% accuracy, significantly outperforming the raw latent code $z$ from VAE-GAN (97.44%).
  • Identity (ID) Classification: The content code $z_c$ achieved 98.59% accuracy, proving it retained identity information better than the original entangled latent space.

Limitations and Future Work

The paper does not explicitly discuss limitations, but a few are inherent to the design:

  • Dependency on Classifiers: The quality of disentanglement is strictly bounded by the accuracy of the pre-trained classifier $C$. If the classifier is biased or inaccurate, the decomposition will fail.
  • Generator Capacity: The method can only manipulate attributes that the pre-trained generator is already capable of producing. It cannot "invent" new features not present in the original training data.
  • Future Directions: The "Beyond" in the title hints at applying this to other domains like text-to-image or cross-domain translation, though these were not explored in this specific paper.

Summary updated

Abstract

Representation disentanglement aims at learning interpretable features, so that the output can be recovered or manipulated accordingly. While existing works like infoGAN and AC-GAN exist, they choose to derive disjoint attribute code for feature disentanglement, which is not applicable for existing/trained generative models. In this paper, we propose a decomposition-GAN (dec-GAN), which is able to achieve the decomposition of an existing latent representation into content and attribute features. Guided by the classifier pre-trained on the attributes of interest, our dec-GAN decomposes the attributes of interest from the latent representation, while data recovery and feature consistency objectives enforce the learning of our proposed method. Our experiments on multiple image datasets confirm the effectiveness and robustness of our dec-GAN over recent representation disentanglement models.
BibTeX
@inproceedings{chen2021representation,
  author = {Shang-Fu Chen and Jia-Wei Yan and Ya-Fan Su and Yu-Chiang Frank Wang},
  title = {{Representation Decomposition For Image Manipulation And Beyond}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506015},
}
Figure