SPA-GAN: Spatial Attention GAN for Image-to-Image Translation

Hajar Emami, Majid Moradi Aliabadi, Ming Dong, Ratna Babu ChinnamIEEE Trans. Multimedia 2021 · DOI

TL;DR

This paper introduces SPA-GAN, a novel Generative Adversarial Network (GAN) architecture for image-to-image translation that directly incorporates a spatial attention mechanism into the discriminator. By allowing the discriminator to highlight discriminative regions, SPA-GAN guides the generator to focus on these key areas, leading to more realistic and accurate translations, especially for object-centric tasks.

What Problem Does This Paper Address?

Image-to-image translation aims to learn a mapping between images from a source domain and a target domain. This is useful for tasks like colorization, super-resolution, and domain adaptation. While many methods require paired training data, which is often scarce, unsupervised approaches have emerged. A key challenge in unsupervised image translation is accurately identifying and transforming specific objects or regions of interest within an image, rather than just altering the overall appearance. Existing attention-based methods often rely on separate attention networks or require additional supervision (like segmentation masks), which can be computationally expensive and limit applicability. This paper addresses the need for a more integrated and efficient attention mechanism within GANs for image-to-image translation, particularly for tasks where precise object transformation is crucial.

Methods

The core innovation of SPA-GAN is the direct integration of a spatial attention mechanism into the discriminator of a GAN. Instead of just classifying an image as real or fake, the discriminator in SPA-GAN also generates "spatial attention maps." These maps highlight the regions within an image that the discriminator finds most discriminative for its classification task.

Here's a breakdown of the key components:

  • Spatial Attention from Discriminator: The discriminator learns to produce attention maps (ADx) that indicate the importance of different spatial locations. These maps are derived from the absolute values of activation maps in a specific layer of the discriminator. The paper uses mid-level attention maps, typically from the second-to-last layer, which are found to correlate well with discriminative object parts.
  • Generator Guidance: These spatial attention maps are then fed back to the generator. By applying an element-wise product with the input image, the generator is guided to focus its translation efforts on these discriminative regions, leading to more accurate and detailed transformations.
  • Feature Map Loss: To further preserve domain-specific features during translation, SPA-GAN introduces an additional "feature map loss." This loss penalizes differences between feature maps in the first layer of the decoders of the generator and its inverse mapping. This encourages the generators to maintain high-level semantic abstractions of the original objects.
  • Modified Cycle Consistency Loss: The standard cycle consistency loss is adapted to incorporate the attended input samples, ensuring that the cycle translation process also benefits from the attention mechanism.

SPA-GAN is designed to be lightweight, not requiring separate attention networks or external supervision like segmentation labels, making it more efficient and broadly applicable.

Datasets

The paper evaluates SPA-GAN on several benchmark datasets for image-to-image translation:

  • Object-centric translation:
    • Horse → Zebra
    • Apple ↔ Orange
    • Lion → Tiger
  • Holistic image translation:
    • Winter → Summer
    • Gender conversion (Facescrub dataset)
    • GTA → Cityscapes

Key Insights and Ideas

  • Discriminator as an Attention Oracle: The most significant insight is leveraging the discriminator's learned discriminative features to guide the generator. Instead of relying on explicit object detection or segmentation, the discriminator implicitly learns what makes an image "real" or "fake" in a domain-specific way, and this knowledge is directly transferable to the generator.
  • Focusing on "What Matters": By explicitly attending to discriminative regions, SPA-GAN can better handle translations where object shape and texture are critical, as opposed to methods that might only change colors or struggle with background consistency.
  • Lightweight and Efficient Attention: The integration of attention directly into the GAN architecture, without additional networks or supervision, makes SPA-GAN computationally more efficient and easier to implement compared to prior attention-based methods.
  • Feature Map Loss for Semantic Preservation: The feature map loss is crucial for maintaining the high-level semantic content of objects, preventing them from becoming overly blurry or losing their essential characteristics during translation.

Key Results

  • Quantitative Performance: SPA-GAN consistently achieved the lowest Kernel Inception Distance (KID) scores on target-only evaluations across multiple datasets, indicating more realistic generated images. It also demonstrated superior classification accuracy on generated images, suggesting they are more representative of the target domain.
  • Qualitative Performance: Visual results show that SPA-GAN excels at translating objects with significant shape and texture differences (e.g., apples to oranges, horses to zebras). It is more robust to shape changes and better preserves object integrity compared to methods like CycleGAN, DRIT, MUNIT, and AGGAN. For holistic translations, SPA-GAN effectively captures domain-specific changes in ground and trees (Winter-Summer) and facial features (gender conversion).
  • User Studies: Human perceptual studies confirmed SPA-GAN's superiority, with participants consistently preferring SPA-GAN's translations over those from other state-of-the-art methods.

Limitations and Future Work

  • Limitations:
    • While SPA-GAN performs well, the paper notes that it doesn't always achieve the smallest KID when computed across both source and target domains. This suggests that for datasets with less distinct backgrounds, other methods might produce images that are statistically closer to both domains, even if less visually realistic for the target.
  • The paper focuses on unsupervised settings, and its performance in fully supervised scenarios is not explicitly detailed.
  • Future Work:
    • The authors suggest exploring other applications of SPA-GAN and further investigating its effectiveness on more complex datasets.
  • The paper's success with spatial attention could inspire further research into integrating different types of attention mechanisms within GANs for various image generation and translation tasks.

Summary updated

BibTeX
@article{emami2021spa,
  author = {Hajar Emami and Majid Moradi Aliabadi and Ming Dong and Ratna Babu Chinnam},
  title = {{SPA-GAN: Spatial Attention GAN for Image-to-Image Translation}},
  journal = {IEEE Trans. Multimedia},
  year = {2021},
  doi = {10.1109/tmm.2020.2975961},
}
Figure