InstaGAN: Instance-aware Image-to-Image Translation

Sangwoo Mo, Minsu Cho, Jinwoo Shin

Abstract

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases.

Introduction

Unsupervised image-to-image (I2I) translation, popularized by models like CycleGAN, has been remarkably successful at swapping textures, such as turning a horse into a zebra. However, these models often struggle when the task requires a significant change in shape or when an image contains multiple instances of a target object. For example, turning pants into a skirt often leads to artifacts where the model confuses the garment with the background. InstaGAN addresses this by incorporating instance-level information (segmentation masks) into the generative process, allowing for precise shape transfiguration while preserving the background.

Methodology

InstaGAN introduces three primary innovations to the standard GAN framework to achieve instance awareness:

  1. Instance-Augmented Architecture: The model processes an image alongside a set of instance masks. It is designed to be permutation-invariant, meaning the order of masks does not affect the output. This is achieved by aggregating instance features into a global set feature via summation.
  2. Context Preserving Loss ($L_{ctx}$): To prevent background distortion, this loss function encourages the network to learn an identity function for any pixel outside the target instances. This ensures that only the intended objects are modified.
  3. Sequential Mini-Batch Translation: To handle images with many instances within limited GPU memory, InstaGAN translates subsets of instances sequentially. This approach also acts as a form of data augmentation, helping the model generalize by seeing partially translated intermediate images during training.

Key Results and Insights

InstaGAN was evaluated on challenging datasets like Clothing Co-Parsing (CCP) and MS COCO, which require significant shape deformation and multi-instance handling.

Comparison of InstaGAN with CycleGAN on fashion (jeans to skirt) and animal (sheep to giraffe) translation tasks.

Visual Quality and Background Integrity

Qualitative results demonstrate that while CycleGAN often produces "ghostly" artifacts or distorted backgrounds, InstaGAN produces clean shape transformations. For instance, it can generate the triangular flare of a skirt or the long neck of a giraffe while keeping the original background pixels identical. This is a direct result of the model knowing exactly where an object is and, crucially, where it isn't.

Quantitative Performance

To quantify success, the researchers used a pre-trained VGG-16 classifier to verify if translated images were recognizable as the target class. In a pants-to-shorts task, InstaGAN achieved 23.2% accuracy, significantly outperforming CycleGAN's 8.5%. The model also proved robust enough to use masks predicted by Mask R-CNN at test time, rather than relying solely on ground-truth data.

Conclusion

InstaGAN represents a significant advancement in I2I translation by making the process instance-aware. While its performance is linked to the quality of the input segmentation masks, it provides a powerful framework for complex shape transfiguration. Future work could extend this approach to video translation to maintain consistency across frames or to other domains involving set-based entities.

Summary updated

Abstract

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan
BibTeX
@inproceedings{mo2019instagan,
  author = {Sangwoo Mo and Minsu Cho and Jinwoo Shin},
  title = {{InstaGAN: Instance-aware Image-to-Image Translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=VP-keeX0TG},
}

Loading related content...

Figure