VQBB: Image-to-image Translation with Vector Quantized Brownian Bridge

Bo Li, Kaitao Xue, Bin Liu, Yu-Kun LaiarXiv 2022 · arXiv · OpenReview

TL;DR

This paper introduces BBDM, a novel image-to-image translation method that leverages the Brownian Bridge diffusion process. Unlike previous diffusion models that treat translation as conditional generation, BBDM models it as a direct stochastic process between domains, leading to improved fidelity and diversity.

What Problem Does This Paper Address?

Image-to-image translation, the task of transforming an image from one domain to another (e.g., sketch to photo, semantic layout to realistic image), is a fundamental problem in computer vision. While Generative Adversarial Networks (GANs) like Pix2Pix and CycleGAN have been popular, they often struggle with training instability and generating diverse outputs, typically producing one-to-one mappings. Diffusion models have shown promise for high-quality image synthesis and diversity, but existing conditional diffusion models for image-to-image translation still face challenges. They often treat translation as a conditional generation task, injecting the source image's features into the reverse diffusion process. This can lead to a gap between the input and output domains and a lack of theoretical guarantees for achieving the desired conditional distribution, especially when domains are significantly different.

Methods

The core innovation of BBDM is its use of the Brownian Bridge diffusion process for image-to-image translation. Instead of the standard diffusion process that gradually adds noise to data until it reaches a pure Gaussian distribution, a Brownian Bridge process is conditioned on both a starting point (the source image's latent representation) and an ending point (the target image's latent representation). This allows BBDM to directly model the transformation between two domains as a stochastic bridge.

To improve efficiency and generalization, BBDM performs the diffusion process in the latent space of a pre-trained VQGAN (Vector Quantized Generative Adversarial Network), similar to Latent Diffusion Models (LDMs). However, BBDM differs fundamentally in how it models the translation. In the reverse diffusion process, the target domain image (or its latent representation) is set as the final state of the Brownian Bridge, not as a conditional input to the neural network at each step. This avoids the issues associated with conditional input leverage in previous methods. The paper also proposes a novel variance schedule for the Brownian Bridge process to prevent issues with extremely large variances at middle steps.

Datasets

The paper evaluates BBDM on several standard image-to-image translation tasks and datasets:

  • CelebAMask-HQ: Used for semantic synthesis.
  • edges2shoes and edges2handbags: Used for sketch-to-photo translation.
  • faces2comics: Used for style transfer.
  • VisualGENOME and CelebAMask-HQ: Used for inpainting, colorization, and face-to-label tasks to demonstrate generalization.

Key Insights and Ideas

  • Brownian Bridge as a natural fit for translation: Modeling image-to-image translation as a Brownian Bridge process, which is inherently tied to start and end points, provides a more direct and theoretically grounded approach than conditional generation.
  • Latent space diffusion for efficiency: Performing diffusion in the latent space of a VQGAN significantly speeds up training and inference while maintaining high-quality results.
  • Avoiding conditional input in reverse process: By setting the target domain as the endpoint of the Brownian Bridge and not using it as a direct conditional input to the denoising network, BBDM circumvents issues of domain gap and conditional information leverage seen in prior diffusion-based methods.
  • Controllable diversity: The proposed variance schedule allows for tuning the sampling diversity by scaling a parameter, offering a trade-off between fidelity and diversity.

Key Results

  • Competitive Performance: BBDM achieves competitive results on various benchmarks, outperforming state-of-the-art GAN-based and diffusion-based methods in terms of FID (Fréchet Inception Distance) on the CelebAMask-HQ dataset.
  • Improved Diversity: The method demonstrates the ability to generate diverse samples, as evidenced by qualitative results and quantitative diversity metrics.
  • Generalization: BBDM shows good generalization capabilities across different image-to-image translation tasks, including inpainting, colorization, and face-to-label.
  • User Study: A user study indicated that participants preferred BBDM's results over those of CDE, LDM, and OASIS.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the Brownian Bridge process offers flexibility, the specific variance schedule designed for BBDM might lead to a decrease in quality and fidelity when diversity is excessively increased (by scaling the variance parameter 's').
  • Future Work: The authors suggest exploring the application of their framework to multi-modal tasks, such as text-to-image generation, which could benefit from the flexible nature of the Brownian Bridge process. They also mention that further improvements to BBDM are possible.

Summary updated

Abstract

Image-to-image translation is an important and challenging problem in computer vision. Existing approaches like Pixel2Pixel, DualGAN suffer from the instability of GAN and fail to generate diverse outputs because they model the task as a one-to-one mapping. Although diffusion models can generate images with high quality and diversity, current conditional diffusion models still can not maintain high similarity with the condition image on image-to-image translation tasks due to the Gaussian noise added in the reverse process. To address these issues, a novel Vector Quantized Brownian Bridge(VQBB) diffusion model is proposed in this paper. On one hand, Brownian Bridge diffusion process can model the transformation between two domains more accurate and flexible than the existing Markov diffusion methods. As far as the authors know, it is the first work for Brownian Bridge diffusion process proposed for image-to-image translation. On the other hand, the proposed method improved the learning efficiency and translation accuracy by confining the diffusion process in the quantized latent space. Finally, numerical experimental results validated the performance of the proposed method.
BibTeX
@misc{li2022vqbb,
  author = {Bo Li and Kaitao Xue and Bin Liu and Yu-Kun Lai},
  title = {{VQBB: Image-to-image Translation with Vector Quantized Brownian Bridge}},
  year = {2022},
  eprint = {2205.07680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.07680},
}
Figure