ClsGAN: Selective Attribute Editing Based On Classification Adversarial Network

Liu Ying, Heng Fan, Fuchuan Ni, Jinhai XiangarXiv 2019 · OpenReview

TL;DR

This paper introduces ClsGAN, a novel approach to image attribute editing that balances high-quality image generation with accurate attribute transfer. It achieves this by proposing a new "Tr-resnet" decoder for better information extraction and an "Atta-cls" attribute adversarial classifier to guide the generator by learning from attribute transfer errors.

What Problem Does This Paper Address?

Image attribute editing, the task of modifying specific characteristics of an image (like hair color or age) while preserving others, has seen significant progress with Generative Adversarial Networks (GANs). However, a key challenge remains: generating images that are both photorealistic and accurately reflect the desired attribute changes. Existing encoder-decoder GANs often struggle with this trade-off. While skip-connections improve image quality, they can degrade attribute accuracy by allowing too much of the original image's attribute information to pass through. Conversely, methods that prioritize attribute accuracy might sacrifice visual realism. This paper aims to bridge this gap by developing a model that excels in both aspects.

Methods

ClsGAN introduces two main novel components:

  • Tr-resnet (Upper Convolution Residual Network): This is a new decoder architecture designed to address the limitations of skip-connections in encoder-decoder structures. Unlike standard skip-connections that simply concatenate features, Tr-resnet selectively extracts information from both the source image and the target attribute label. It achieves this by combining the input and output of its residual blocks in a weighted manner, allowing it to better disentangle and utilize relevant information for attribute transfer.
  • Atta-cls (Attribute Adversarial Classifier): To further enhance attribute transfer accuracy, an adversarial classifier is introduced. Inspired by the adversarial nature of GANs, Atta-cls is trained to identify defects in attribute transfer. By feeding both original and generated images to this classifier, it learns to distinguish between correctly and incorrectly transferred attributes. This feedback loop then guides the generator to produce images with more accurate attribute transformations.

The overall architecture involves two encoders (one for attribute, one for content) and the Tr-resnet decoder, forming the generator. The discriminator is composed of both a standard adversarial discriminator and the Atta-cls.

Datasets

The primary dataset used for training and evaluation is CelebA. This dataset contains over 200,000 celebrity face images with 40 facial attributes. The authors selected 13 specific attributes for their experiments, including "Bald," "Bangs," "Eyeglasses," and "Gender." They also mention experiments on a season dataset and a painting dataset for style transfer, but details on these datasets are less extensive.

Key Insights and Ideas

  • The "bottleneck" of skip-connections: The paper highlights that while skip-connections in encoder-decoder architectures improve image quality, they can inadvertently weaken the target attribute information by allowing too much original attribute detail to pass through. This is a crucial insight into a common trade-off in image editing.
  • Selective information fusion: Tr-resnet's approach of selectively combining input and output features within residual blocks is a clever way to manage information flow, allowing for better disentanglement of content and attribute information.
  • Adversarial learning for attribute accuracy: Using an adversarial classifier (Atta-cls) to learn the defects of attribute transfer is a novel way to improve accuracy. Instead of just classifying attributes, it learns to identify errors, providing a more targeted signal to the generator.
  • Attribute continuity: The model is designed to handle continuous attribute values, not just binary (present/absent) attributes, allowing for more nuanced editing.

Key Results

  • Improved Image Quality and Accuracy: ClsGAN demonstrates significant improvements in both image quality (measured by FID) and attribute transfer accuracy compared to state-of-the-art methods like StarGAN, AttGAN, and STGAN. For instance, it achieves a FID of 6.09, outperforming StarGAN's 7.9.
  • High Reconstruction Performance: The reconstructed images show better consistency with original images in terms of background and details.
  • Effective Attribute Transfer: ClsGAN shows strong performance across various attributes, including challenging ones like "Mustache" (0.523 transfer accuracy).
  • Ablation Studies: Experiments confirm the effectiveness of both Tr-resnet and Atta-cls. Replacing Tr-resnet with simpler convolutional networks leads to degraded performance, and using a standard classifier instead of Atta-cls results in less realistic images with artifacts.

Limitations and Future Work

  • Artifacts in Large-Range Editing: The authors acknowledge that for very large-range texture and color editing, some artifacts can still occur, and balancing image quality with attribute transformation remains a complex challenge.
  • Potential for Further Exploration: ClsGAN is presented as a promising model that warrants further exploration and extension, particularly for more complex style transfer tasks. The authors suggest that the model's ability to handle attribute continuity and its overall effectiveness make it a good foundation for future research.

Summary updated

Abstract

Attribution editing has achieved remarkable progress in recent years owing to the encoder-decoder structure and generative adversarial network (GAN). However, it remains challenging in generating high-quality images with accurate attribute transformation. Attacking these problems, the work proposes a novel selective attribute editing model based on classification adversarial network (referred to as ClsGAN) that shows good balance between attribute transfer accuracy and photo-realistic images. Considering that the editing images are prone to be affected by original attribute due to skip-connection in encoder-decoder structure, an upper convolution residual network (referred to as Tr-resnet) is presented to selectively extract information from the source image and target label. In addition, to further improve the transfer accuracy of generated images, an attribute adversarial classifier (referred to as Atta-cls) is introduced to guide the generator from the perspective of attribute through learning the defects of attribute transfer images. Experimental results on CelebA demonstrate that our ClsGAN performs favorably against state-of-the-art approaches in image quality and transfer accuracy. Moreover, ablation studies are also designed to verify the great performance of Tr-resnet and Atta-cls.
BibTeX
@misc{ying2019clsgan,
  author = {Liu Ying and Heng Fan and Fuchuan Ni and Jinhai Xiang},
  title = {{ClsGAN: Selective Attribute Editing Based On Classification Adversarial Network}},
  year = {2019},
  eprint = {1910.11764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.11764},
}
Figure