Attention-GAN for Object Transfiguration in Wild Images

Xinyuan Chen, Chang Xu, Xiaokang Yang, Dacheng Tao

Introduction

In unpaired image-to-image translation, "object transfiguration" aims to change a specific object—such as turning a horse into a zebra—while keeping the rest of the image unchanged. While models like CycleGAN have popularized this task, they often suffer from a major flaw: the accidental alteration of backgrounds. Because a single generator must simultaneously identify where the object is and how to translate it, it frequently hallucinates changes in the environment, such as turning green grass brown or blue water yellow.

Examples of object transfiguration: horse to zebra and zebra to horse.

The Attention-GAN Architecture

To address these limitations, Attention-GAN decouples the tasks of object detection and image translation. The generator is split into two distinct networks that work together via a layered compositing operation:

  1. Attention Network: This network predicts a spatial attention map (a mask with values from 0 to 1) to isolate the region of interest.
  2. Transformation Network: This network generates a fully translated version of the input image.

The final image is constructed by taking the transformed image where the attention mask is high and retaining the original image where the mask is low. This ensures that the background remains pristine.

Detailed internal architecture of the generator, showing the Attention and Transformation networks and the final compositing step.

Novel Loss Functions

Training the attention network without direct supervision requires specific constraints. The authors introduce two key losses:

  • Attention Cycle-Consistent Loss: The attention mask predicted for the original image must match the mask predicted for the translated image.
  • Attention Sparse Loss: This penalizes large masks, forcing the network to focus strictly on the object and preventing it from selecting the entire image.

The model can also be trained in a supervised manner using ground-truth segmentation masks (e.g., from MSCOCO) to further refine the attention mechanism.

Key Results and Insights

Attention-GAN significantly outperforms standard CycleGAN in background preservation. On the horse → zebra task, Attention-GAN achieved a background PSNR of 22.26 and SSIM of 0.90, compared to CycleGAN's 18.18 and 0.67.

Human perceptual studies further validate these results. In Amazon Mechanical Turk A/B tests, humans preferred Attention-GAN's outputs in 92% of horse/zebra comparisons and 70% of tiger/leopard comparisons. The model's success is largely attributed to its ability to ignore complex backgrounds—such as riders or specific terrain—that typically confuse standard models.

Conclusion

By decoupling attention from transformation, Attention-GAN provides a robust solution for object transfiguration in wild images. The inclusion of sparsity constraints and attention cycle-consistency allows the model to achieve high-quality results even in unsupervised settings, effectively protecting the original context of the image.

Summary updated

BibTeX
@inproceedings{chen2018attention,
  author = {Xinyuan Chen and Chang Xu and Xiaokang Yang and Dacheng Tao},
  title = {{Attention-GAN for Object Transfiguration in Wild Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01216-8\_11},
}
Figure