Abstract
This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model. The use of generative cooperative network enables maximum likelihood learning of the domain model by MCMC teaching, where the energy-based model seeks to fit the data distribution of domain and distills its knowledge to the latent variable model via MCMC. Specifically, in the MCMC teaching process, the latent variable model parameterized by an encoder-decoder maps examples from the source domain to the target domain, while the energy-based model further refines the mapped results by Langevin revision such that the revised results match to the examples in the target domain in terms of the statistical properties, which are defined by the learned energy function. For the purpose of building up a correspondence between two unpaired domains, the proposed framework simultaneously learns a pair of cooperative networks with cycle consistency, accounting for a two-way translation between two domains, by alternating MCMC teaching. Experiments show that the proposed framework is useful for unsupervised image-to-image translation and unpaired image sequence translation.
Introduction
Unsupervised cross-domain translation—the task of mapping images from one domain to another without paired examples—is a fundamental challenge in computer vision. While Generative Adversarial Networks (GANs) have achieved significant success in this area, they are often plagued by training instabilities, such as mode collapse, and the adversarial objective does not explicitly model the target data distribution.
This paper introduces Cycle-Consistent Cooperative Networks (CycleCoopNets), a framework that shifts from adversarial competition to a cooperative maximum likelihood approach. By leveraging Energy-Based Models (EBMs) and Latent Variable Models (LVMs), the authors provide a more stable and theoretically grounded method for unsupervised translation.
Methodology: The Cooperative Network
The fundamental building block of this framework is the Cooperative Network (CoopNet). A CoopNet integrates two distinct models:
- Latent Variable Model (LVM): Acting as a "fast thinker," the LVM (often an encoder-decoder) provides an initial, efficient mapping from the source domain to the target domain.
- Energy-Based Model (EBM): Acting as a "slow thinker," the EBM defines an energy function over the target domain. Lower energy corresponds to higher probability density, representing images that more closely match the target domain's statistics.
MCMC Teaching
The models are trained via a process called MCMC Teaching. Instead of a discriminator, the EBM uses Langevin dynamics (a type of MCMC sampling) to "revise" the initial outputs of the LVM. This revision process nudges the generated pixels toward the manifold of the target domain as defined by the EBM's energy function. The LVM then learns to minimize the reconstruction error between its initial guess and the EBM's refined version, effectively distilling the EBM's knowledge.
To handle unpaired data, the authors employ two CoopNets (one for each translation direction) linked by a cycle consistency loss. This ensures the mappings are bijections that preserve the underlying structure of the source images.
Experimental Results
The researchers evaluated CycleCoopNets across diverse tasks, including object transfiguration (Apple $\Leftrightarrow$ Orange), season transfer, and architectural label mapping.
Qualitative Performance
Visual results demonstrate that CycleCoopNets produce more detailed and realistic translations than standard baselines like CycleGAN and UNIT. Unlike GANs, the learned EBM allows for iterative refinement at inference time, enabling the model to "polish" its outputs through Langevin dynamics.
Quantitative Superiority
On the Apple $\Leftrightarrow$ Orange benchmark, CycleCoopNets achieved a Fréchet Inception Distance (FID) score of 154.58, outperforming CycleGAN's 160.78. Lower FID scores indicate that the generated images are both more realistic and more diverse. The framework also showed exceptional stability, avoiding the mode collapse often seen in adversarial training. For video sequences, the model successfully maintained temporal consistency in facial expression transfers without the jitter common in frame-by-frame GAN approaches.
Conclusion
CycleCoopNets represent a significant step toward stable, likelihood-based unsupervised translation. By replacing adversarial training with cooperative MCMC teaching, the framework achieves high-fidelity results with better theoretical properties. While the computational cost of MCMC sampling makes training approximately three times slower than GAN-based methods, the benefits in stability and the ability to refine outputs at test time make it a compelling alternative for complex translation tasks.
Summary updated