Guided Image-to-Image Translation by Discriminator-Generator Communication

Yuanjiang Cao, Lina Yao, Le Pan, Quan Z. Sheng, Xiaojun Chang

TL;DR

This paper reframes Generative Adversarial Network (GAN) training for image-to-image (I2I) translation as a Partially-observed Markov Decision Process (POMDP). To overcome the "information insufficiency" inherent in this setup, the authors introduce a direct communication channel that allows discriminators to send continuous guidance vectors to generators, significantly improving the quality and texture of translated images.

What Problem Does This Paper Address?

Image-to-image (I2I) translation—tasks like turning a horse into a zebra or an apple into an orange—often relies on GANs. In the standard setup, the discriminator acts as a critic, seeing both real and fake images to learn the target domain's distribution. However, the generator is "partially blind": it only sees source images and its own outputs, relying solely on the scalar adversarial loss signal to understand what it's doing wrong.

The authors argue that this creates information insufficiency. From a decision-making perspective, the discriminator operates in a full Markov Decision Process (MDP), while the generator is stuck in a POMDP. This gap makes training unstable and often leads to translated images with poor textures or artifacts because the generator lacks a nuanced understanding of the target domain's features.

Fig. 1. The defined Partially-observed Markov Decision Process (POMDP) based on GAN. D, G, X represent a discriminator, a generator and a batch of images from a dataset, respectively. s is state, sr is an image from the real domain, and sf is an image from the fake domain. r is the reward for an agent, and a is the action for the agent. The black dashed arc in this figure shows the comparison of the state source between the MDP of the discriminator and the POMDP of the generator.

Methods

The core innovation is the introduction of a communication channel from the discriminator to the generator.

  1. Guidance Vectors: Instead of just outputting a real/fake score, the discriminator ($D_g$) produces a continuous "guidance vector" ($g$). This vector acts as a probe, carrying salient feature information from the discriminator's internal representations.
  2. Generator Integration: The generator ($G$) is modified to take both the input image ($x$) and the guidance vector ($g$) as inputs: $\hat{x} = G(x, g)$. The authors explored two merging strategies: simple concatenation with a neural mapping and a more complex self-attention mechanism.
  3. Two-Step Training: To prevent training instability caused by using the same discriminator for both guidance and loss calculation in a single pass, the authors developed a two-step training algorithm. This ensures the discriminator is updated effectively for both its roles.
  4. Regularization: They introduced an $L_1$ regularization loss on the guidance vectors to ensure they reflect meaningful saliency information about the images.
Integration of the communication module into the CycleGAN framework.

Datasets

The model was evaluated on several standard unpaired I2I benchmarks:

  • Apple $\leftrightarrow$ Orange: A classic object transfiguration dataset.
  • Horse $\leftrightarrow$ Zebra: Used to test the model's ability to handle complex textures and patterns.
  • Lion $\leftrightarrow$ Tiger: A dataset reproduced by the authors using images from ImageNet to further validate object transfiguration.

Key Insights and Ideas

  • GANs as POMDPs: The most significant conceptual contribution is viewing the generator's lack of information as a POMDP problem. This allows the authors to borrow solutions from multi-agent reinforcement learning (MARL), where communication is a standard way to solve partial observability.
  • Information Bottleneck: By using a relatively small guidance vector, the discriminator is forced to compress its knowledge into the most "useful" features, filtering out noise before passing it to the generator.
  • Input vs. Output Domain Guidance: The authors found that guidance from the input domain discriminator helps balance performance between the two translation directions (e.g., apple-to-orange vs. orange-to-apple), whereas guidance from the output domain discriminator (the one forming the adversarial game) provides the most direct performance boost.

Key Results

The proposed method was compared against several heavyweights, including CycleGAN, MUNIT, DRIT, and AttentionGAN.

  • Quantitative: Using the Kernel Inception Distance (KID)—a metric that measures the discrepancy between image distributions—the authors' model achieved the best or second-best results across all tested domains. For example, in the horse → zebra task, their model achieved a KID of $1.61 \pm 0.52$, significantly lower (better) than CycleGAN's $3.94 \pm 0.41$.
  • Qualitative: Visually, the model produced much cleaner textures. In the apple-to-orange task, it successfully captured fine details like the "green dots" on certain apple varieties that baselines often blurred or ignored. It also showed superior background preservation, keeping fingers and leaves sharp while transforming the fruit.
Fig. 3. Qualitative experiment results on apple2orange dataset, compared with multiple models
Fig. 4. Qualitative experiment results on horse2zebra dataset, compared with multiple models

Limitations and Future Work

  • Complexity of Multi-Guidance: While the authors explored merging guidance from multiple discriminators (using weighted averages or bi-GRUs), these more complex schemes often performed worse than single-discriminator guidance. They suspect this is due to increased optimization difficulty and gradients being harder to propagate.
  • Computational Overhead: The two-step training and the additional guidance modules add computational cost, though the authors describe the architectural changes as "slight."
  • Future Directions: The authors suggest that exploring more sophisticated communication protocols from MARL or finding better ways to fuse multi-discriminator information could further push the boundaries of I2I translation quality.

Summary updated

Abstract

The goal of Image-to-image (I2I) translation is to transfer an image from a source domain to a target domain, which has recently drawn increasing attention. One major branch of this research is to formulate I2I translation based on Generative Adversarial Network (GAN). As a zero-sum game, GAN can be reformulated as a Partially-observed Markov Decision Process (POMDP) for generators, where generators cannot access full state information of their environments. This formulation illustrates the information insufficiency in the GAN training. To mitigate this problem, we propose to add a communication channel between discriminators and generators. We explore multiple architecture designs to integrate the communication mechanism into the I2I translation framework. To validate the performance of the proposed approach, we have conducted extensive experiments on various benchmark datasets. The experimental results confirm the superiority of our proposed method.
BibTeX
@article{cao2024guided,
  author = {Yuanjiang Cao and Lina Yao and Le Pan and Quan Z. Sheng and Xiaojun Chang},
  title = {{Guided Image-to-Image Translation by Discriminator-Generator Communication}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2023.3282869},
}
Figure