TL;DR
This paper reframes Generative Adversarial Network (GAN) training for image-to-image (I2I) translation as a Partially-observed Markov Decision Process (POMDP). To overcome the "information insufficiency" inherent in this setup, the authors introduce a direct communication channel that allows discriminators to send continuous guidance vectors to generators, significantly improving the quality and texture of translated images.
What Problem Does This Paper Address?
Image-to-image (I2I) translation—tasks like turning a horse into a zebra or an apple into an orange—often relies on GANs. In the standard setup, the discriminator acts as a critic, seeing both real and fake images to learn the target domain's distribution. However, the generator is "partially blind": it only sees source images and its own outputs, relying solely on the scalar adversarial loss signal to understand what it's doing wrong.
The authors argue that this creates information insufficiency. From a decision-making perspective, the discriminator operates in a full Markov Decision Process (MDP), while the generator is stuck in a POMDP. This gap makes training unstable and often leads to translated images with poor textures or artifacts because the generator lacks a nuanced understanding of the target domain's features.
Methods
The core innovation is the introduction of a communication channel from the discriminator to the generator.
- Guidance Vectors: Instead of just outputting a real/fake score, the discriminator ($D_g$) produces a continuous "guidance vector" ($g$). This vector acts as a probe, carrying salient feature information from the discriminator's internal representations.
- Generator Integration: The generator ($G$) is modified to take both the input image ($x$) and the guidance vector ($g$) as inputs: $\hat{x} = G(x, g)$. The authors explored two merging strategies: simple concatenation with a neural mapping and a more complex self-attention mechanism.
- Two-Step Training: To prevent training instability caused by using the same discriminator for both guidance and loss calculation in a single pass, the authors developed a two-step training algorithm. This ensures the discriminator is updated effectively for both its roles.
- Regularization: They introduced an $L_1$ regularization loss on the guidance vectors to ensure they reflect meaningful saliency information about the images.
Datasets
The model was evaluated on several standard unpaired I2I benchmarks:
- Apple $\leftrightarrow$ Orange: A classic object transfiguration dataset.
- Horse $\leftrightarrow$ Zebra: Used to test the model's ability to handle complex textures and patterns.
- Lion $\leftrightarrow$ Tiger: A dataset reproduced by the authors using images from ImageNet to further validate object transfiguration.
Key Insights and Ideas
- GANs as POMDPs: The most significant conceptual contribution is viewing the generator's lack of information as a POMDP problem. This allows the authors to borrow solutions from multi-agent reinforcement learning (MARL), where communication is a standard way to solve partial observability.
- Information Bottleneck: By using a relatively small guidance vector, the discriminator is forced to compress its knowledge into the most "useful" features, filtering out noise before passing it to the generator.
- Input vs. Output Domain Guidance: The authors found that guidance from the input domain discriminator helps balance performance between the two translation directions (e.g., apple-to-orange vs. orange-to-apple), whereas guidance from the output domain discriminator (the one forming the adversarial game) provides the most direct performance boost.
Key Results
The proposed method was compared against several heavyweights, including CycleGAN, MUNIT, DRIT, and AttentionGAN.
- Quantitative: Using the Kernel Inception Distance (KID)—a metric that measures the discrepancy between image distributions—the authors' model achieved the best or second-best results across all tested domains. For example, in the
horse → zebratask, their model achieved a KID of $1.61 \pm 0.52$, significantly lower (better) than CycleGAN's $3.94 \pm 0.41$. - Qualitative: Visually, the model produced much cleaner textures. In the apple-to-orange task, it successfully captured fine details like the "green dots" on certain apple varieties that baselines often blurred or ignored. It also showed superior background preservation, keeping fingers and leaves sharp while transforming the fruit.
Limitations and Future Work
- Complexity of Multi-Guidance: While the authors explored merging guidance from multiple discriminators (using weighted averages or bi-GRUs), these more complex schemes often performed worse than single-discriminator guidance. They suspect this is due to increased optimization difficulty and gradients being harder to propagate.
- Computational Overhead: The two-step training and the additional guidance modules add computational cost, though the authors describe the architectural changes as "slight."
- Future Directions: The authors suggest that exploring more sophisticated communication protocols from MARL or finding better ways to fuse multi-discriminator information could further push the boundaries of I2I translation quality.
Summary updated