Abstract
Current unsupervised image-to-image translation techniques struggle to focus their attention on individual objects without altering the background or the way multiple objects interact within a scene. Motivated by the important role of attention in human perception, we tackle this limitation by introducing unsupervised attention mechanisms that are jointly adversarialy trained with the generators and discriminators. We demonstrate qualitatively and quantitatively that our approach is able to attend to relevant regions in the image without requiring supervision, and that by doing so it achieves more realistic mappings compared to recent approaches.
Introduction
Unsupervised image-to-image (I2I) translation aims to map images between domains (e.g., from horses to zebras) without paired training data. While models like CycleGAN have popularized this field, they often suffer from a "global change" problem. Because these models lack object-specific focus, they frequently alter the background to match the target domain's distribution—for instance, adding zebra-like patterns to the grass in a horse-to-zebra translation. This occurs because the discriminator evaluates the entire image, incentivizing the generator to modify every pixel to satisfy the target domain's global statistics.
Methodology: Unsupervised Attention
The proposed framework explicitly models attention to isolate object translation from background preservation.
Architecture
The core architecture integrates two primary components:
- Attention Networks ($A_S, A_T$): These networks generate continuous attention masks (values in $[0, 1]$) for the source and target domains.
- Attention-Guided Generators: The generator's output is blended with the original input using the attention mask. The final translated image $s'$ is computed as: $$s' = (mask \odot translated\_foreground) + ((1 - mask) \odot original\_background)$$
Two-Stage Discriminator Training
A critical innovation is the two-stage training process. Initially, the discriminator observes the full image to help the attention networks mature. After 30 epochs, the discriminator is restricted to the masked regions. This forces the generator to focus exclusively on making the target object realistic without "cheating" by altering the background.
Experimental Results
The model was evaluated on benchmarks including Apple $\leftrightarrow$ Orange, Horse $\leftrightarrow$ Zebra, and Lion $\leftrightarrow$ Tiger.
Quantitative Performance
Using the Kernel Inception Distance (KID) metric (lower is better), the proposed method consistently outperformed baselines:
- Apple $\to$ Orange: 6.44 (vs. CycleGAN's 8.48).
- Horse $\to$ Zebra: 8.87 (vs. CycleGAN's 11.44).
Qualitative Comparison
Qualitatively, the attention-guided approach produces significantly cleaner results. Unlike baseline models that often introduce artifacts into the background, this method maintains identical background pixels to the source, resulting in more photorealistic translations.
Discussion and Insights
The success of this approach stems from several key factors:
- Discriminator as Teacher: The attention network learns by identifying regions that are most discriminative for the target domain.
- Cycle-Consistency: The cycle-consistency loss not only preserves content but also sharpens the attention masks over time, making them more binary.
- Robustness to "Empty" Images: If a source object is absent (e.g., a landscape without horses), the attention mask remains zero, leaving the image untouched—a scenario where standard models often fail.
Limitations
The primary limitation is geometric robustness. While the model excels at texture and pattern translation (e.g., stripes), it struggles with significant structural changes, such as altering a silhouette to turn a horse into a bird. Future work may explore spatial transformer networks to accommodate these structural deformations.
Summary updated