InstaGAN: Instance-aware Image-to-Image Translation

Sangwoo Mo, Minsu Cho, Jinwoo Shin

TL;DR

Unpaired image-to-image translation models like CycleGAN struggle when tasks require drastic shape modifications across multiple objects in cluttered backgrounds. InstaGAN solves this by incorporating instance segmentation masks via a permutation-invariant/equivariant set architecture, a background-protecting context-preserving loss, and a sequential mini-batch training strategy. The result is realistic multi-object transfiguration (e.g., pants to skirts, sheep to giraffes) while leaving the surrounding context intact.

What Problem Does This Paper Address?

Traditional unsupervised image-to-image translation frameworks (such as CycleGAN) excel at texture or color style shifts (e.g., horses to zebras, summer to winter). However, they frequently break down in two common scenarios:

  1. When multiple target instances appear in a single image.
  2. When the translation requires dramatic changes in geometric shape (such as turning pants into a skirt or sheep into long-necked giraffes).

Prior mask- or attention-guided models simply cropped target objects and pasted them back onto the background, which inherently prevents instances from expanding or changing their boundary contours. Furthermore, translating multiple instances simultaneously causes standard networks to merge distinct objects, hallucinate artifacts, or mutate the background.

Methods

InstaGAN extends the two-generator, two-discriminator cycle-consistency setup to map jointly between image-instance pairs: $\mathcal{X} \times \mathcal{A} \leftrightarrow \mathcal{Y} \times \mathcal{B}$, where $\mathcal{A}$ and $\mathcal{B}$ are sets of instance binary segmentation masks. Its core technical components include:

  • Set-Based Architecture: The network must handle an arbitrary number of instances without being sensitive to their ordering. The generator extracts features from both the image ($f_{GX}$) and each individual instance mask ($f_{GA}$). Individual mask features are summed into a permutation-invariant context representation ($\sum_i f_{GA}(a_i)$). Concatenating this sum with the image yields a permutation-invariant translated image, while concatenating it with individual mask features produces permutation-equivariant output masks. The discriminator similarly sums instance features to remain permutation-invariant.
  • Context Preserving Loss ($\mathcal{L}_{\text{ctx}}$): To allow shapes to freely deform while safeguarding non-target regions, InstaGAN introduces a pixel-wise weighted $\ell_1$ loss. The loss applies a weight of 1 only to pixels that belong to the background in both the original and translated masks, penalizing unintended background alterations.
  • Sequential Mini-Batch Training & Inference: Handling many masks simultaneously quickly exhausts GPU memory. InstaGAN divides instance masks into smaller subsets (mini-batches) and translates them iteratively, feeding the intermediate image to the next step. To maintain proper training signals, GAN domain losses are applied to the aggregated set of translated masks, while content and cycle losses supervise individual step transitions. Intermediate activations are detached to keep memory usage constant regardless of instance count.
       Original Image + Instance Masks (a_1, ..., a_N)
                           │
             ┌─────────────┴─────────────┐
             ▼                           ▼
    [Set-Feature Pooling]      [Sequential Mini-Batches]
     Summed mask embeddings     Detached iterative updates
             │                           │
             └─────────────┬─────────────┘
                           ▼
                  InstaGAN Generator
                 (Joint Image + Masks)
                           │
             ┌─────────────┴─────────────┐
             ▼                           ▼
    Adversarial + Cycle Loss    Context Preserving Loss (L_ctx)
   (Domain style & consistency) (Penalizes changes outside masks)

Datasets

The authors evaluated InstaGAN on multi-instance benchmarks:

  • Clothing Co-Parsing (CCP): Fashion images used for clothing transformations (jeans $\leftrightarrow$ skirt, shorts $\leftrightarrow$ pants).
  • Multi-Human Parsing (MHP): Cluttered human parsing scenes with multiple people and overlapping garments.
  • MS COCO: Complex multi-object natural scenes used for non-rigid animal and object transfigurations (e.g., sheep $\leftrightarrow$ giraffe, zebra $\leftrightarrow$ elephant, bird $\leftrightarrow$ zebra, car $\leftrightarrow$ horse).

Key Insights and Ideas

  • Joint Image-Mask Discrimination: Separating image generation and mask generation leads to severe mismatches. Jointly feeding both the image and the mask set to generators and discriminators forces the model to render instances precisely within their predicted boundaries.
  • Deformation via Background Regularization: Instead of constraining instances to stay within fixed bounding boxes, the context preserving loss penalizes alterations outside the combined envelope of the input and output masks. This allows object contours to naturally expand or contract.
  • Size-Ordered Sequential Processing: When partitioning instances into mini-batches, sorting them in decreasing order of spatial area outperformed random ordering. Larger objects establish dominant image structures first, reducing occlusion issues for smaller objects downstream.
  • Sequential Steps as Implicit Augmentation: Generating and penalizing intermediate translated samples exposes the network to partial transformations during training, improving generalization even on test images containing few instances.

Key Results

InstaGAN noticeably outperformed standard CycleGAN (even when CycleGAN's parameter count was doubled for a fair comparison) across quantitative and qualitative benchmarks:

  • Target Classification Scores (CCP): Using a fine-tuned VGG-16 classifier to assess how convincingly translated images reflected the target category:
  • pants $\to$ shorts: InstaGAN achieved 23.2% test accuracy vs. CycleGAN's 8.5%.
  • jeans $\to$ skirt: InstaGAN achieved 60.0% test accuracy vs. CycleGAN's 37.1%.
  • shorts $\to$ pants: InstaGAN reached 76.8% test accuracy vs. CycleGAN's 52.4%.
  • Masked Classification Scores (COCO): Evaluated strictly within target instance boundaries to eliminate background false positives:
  • sheep $\to$ giraffe: InstaGAN reached 78.1% test accuracy vs. CycleGAN's 59.4%.
  • giraffe $\to$ sheep: InstaGAN reached 64.2% test accuracy vs. CycleGAN's 51.2%.
  • Ablations: A baseline trained on a single merged segmentation mask (CycleGAN+Seg) frequently merged multiple adjacent instances into a single blob. InstaGAN's permutation-equivariant design kept individual instances separated and preserved fine layout details (e.g., orientation and gaze direction).

Limitations and Future Work

  • Sequential Step Limits: Chaining too many sequential inference steps can accumulate noise and degrade the strict permutation-invariance of the output.
  • Reliance on Instance Masks: The framework requires instance segmentation annotations during training. While the authors demonstrated that off-the-shelf segmenters (like pix2pix or Mask R-CNN) can supply masks at test time, noisy predicted masks can directly degrade translation fidelity.
  • Future Directions: Extending set-structured conditioning beyond static images to video-to-video translation (such as integrating temporal dynamics via Recycle-GAN) and applying set-based conditioning to domains like cross-lingual natural language generation.

Summary updated

Abstract

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan
BibTeX
@inproceedings{mo2019instagan,
  author = {Sangwoo Mo and Minsu Cho and Jinwoo Shin},
  title = {{InstaGAN: Instance-aware Image-to-Image Translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=VP-keeX0TG},
}

Loading related content...

Figure