Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation

Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, Ying Nian Wu

Abstract: This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model. The use of generative cooperative network enables maximum likelihood learning of the domain model by MCMC teaching, where the energy-based model seeks to fit the data distribution of domain and distills its knowledge to the latent variable model via MCMC. Specifically, in the MCMC teaching process, the latent variable model parameterized by an encoder-decoder maps examples from the source domain to the target domain, while the energy-based model further refines the mapped results by Langevin revision such that the revised results match to the examples in the target domain in terms of the statistical properties, which are defined by the learned energy function. For the purpose of building up a correspondence between two unpaired domains, the proposed framework simultaneously learns a pair of cooperative networks with cycle consistency, accounting for a two-way translation between two domains, by alternating MCMC teaching. Experiments show that the proposed framework is useful for unsupervised image-to-image translation and unpaired image sequence translation.

Introduction

Cross-domain translation—the task of mapping an image from a source domain (e.g., "apples") to a target domain (e.g., "oranges") without paired training data—is a fundamental challenge in computer vision. While unsupervised methods like CycleGAN have achieved significant success, they often suffer from training instabilities such as mode collapse and a lack of internal mechanisms for iterative output refinement.

This research introduces CycleCoopNets, a framework that shifts the paradigm from adversarial training to a cooperative learning strategy. By replacing the discriminator with an Energy-Based Model (EBM) that "teaches" a generator through Markov Chain Monte Carlo (MCMC) sampling, the framework achieves more stable training and provides a built-in mechanism to refine translated images for enhanced realism.

Methodology: Cycle-Consistent Cooperative Networks

The CycleCoopNets architecture consists of two cooperative networks, one for each translation direction ($X \to Y$ and $Y \to X$). Each network integrates two components:

  1. The Latent Variable Model (LVM/Generator): A "fast-thinking" translator that maps an image from the source domain to an initial draft in the target domain.
  2. The Energy-Based Model (EBM): A "slow-thinking" teacher that defines the probability distribution over the target domain based on its statistical properties.

The MCMC Teaching Process

Instead of the binary feedback provided by a GAN discriminator, the EBM uses Langevin dynamics (a type of MCMC sampling) to actively improve the generator's output. The process follows three main stages: the generator creates a draft, the EBM "revises" this draft over several steps to align it with the target domain's statistics, and the generator then updates its parameters to better predict these refined results. To maintain semantic meaning across domains, a cycle consistency loss is enforced, ensuring that an image translated to the target domain and back remains consistent with the original.

Illustration of the alternating MCMC teaching process for unsupervised cross-domain translation.

Experimental Results

The model was evaluated across diverse benchmarks, including object transfiguration (Apple $\leftrightarrow$ Orange), season transfer (Summer $\leftrightarrow$ Winter), and art style transfer (Photos $\leftrightarrow$ Monet/Van Gogh).

Quantitative Performance

CycleCoopNets demonstrates superior performance in distribution matching. On the Apple $\leftrightarrow$ Orange task, the model achieved a Frechet Inception Distance (FID) of 154.58, outperforming CycleGAN (160.78) and UNIT (170.66). A lower FID indicates that the generated images are statistically more similar to the real target distribution. Furthermore, using Domain-invariant Perceptual Distance (DIPD), the authors confirmed that the model preserves the original content's structure more effectively than adversarial baselines.

Qualitative Realism

Qualitatively, CycleCoopNets produces images with more realistic textures and fewer artifacts. A unique advantage of this framework is that at test time, the EBM can be used to further "polish" the generator's output via MCMC refinement, leading to noticeable improvements in visual quality.

Qualitative comparison of unsupervised image-to-image translation (apple to orange and vice versa) against CycleGAN and UNIT.

Discussion and Conclusion

CycleCoopNets offers a robust alternative to GANs for unsupervised translation by leveraging the stability of maximum likelihood estimation. The cooperative interaction between the EBM and LVM not only prevents mode collapse but also enables a unique refinement capability. While MCMC sampling introduces higher computational costs during training compared to standard GANs, the resulting stability and image quality provide a compelling trade-off. Future work may explore applying this "teaching" framework to other modalities, such as cross-domain text or audio translation.

Summary updated

Abstract

This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model. The use of generative cooperative network enables maximum likelihood learning of the domain model by MCMC teaching, where the energy-based model seeks to fit the data distribution of domain and distills its knowledge to the latent variable model via MCMC. Specifically, in the MCMC teaching process, the latent variable model parameterized by an encoder-decoder maps examples from the source domain to the target domain, while the energy-based model further refines the mapped results by Langevin revision such that the revised results match to the examples in the target domain in terms of the statistical properties, which are defined by the learned energy function. For the purpose of building up a correspondence between two unpaired domains, the proposed framework simultaneously learns a pair of cooperative networks with cycle consistency, accounting for a two-way translation between two domains, by alternating MCMC teaching. Experiments show that the proposed framework is useful for unsupervised image-to-image translation and unpaired image sequence translation.
BibTeX
@inproceedings{xie2021learning,
  author = {Jianwen Xie and Zilong Zheng and Xiaolin Fang and Song-Chun Zhu and Ying Nian Wu},
  title = {{Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=luGLg92kMhN},
}
Figure