PROMOTE: Prior-Guided Diffusion Model with Global-Local Contrastive Learning for Exemplar-Based Image Translation

Guojin Zhong, Yihu Guo, Jin Yuan, Qianjun Zhang, Weili Guan, Long Chen

TL;DR

This paper introduces PROMOTE, a novel approach to exemplar-based image translation that leverages hyperbolic geometry for more effective feature learning and a prior-guided diffusion model for improved translation fidelity. By aligning cross-domain image features in hyperbolic space and incorporating structural priors into the diffusion process, PROMOTE significantly enhances the quality and consistency of generated images.

What Problem Does This Paper Address?

Exemplar-based image translation aims to transform an input image (e.g., a sketch or pose) into a realistic image that adopts the style of a separate "exemplar" image. Existing methods often struggle with two main issues:

  1. Poor Domain-Invariant Features: They rely on Euclidean distances to match features between the input and exemplar images. This can be insufficient because Euclidean space doesn't effectively capture the complex relationships and subtle differences between images from different domains (e.g., a sketch vs. a photograph). This leads to low-quality correspondences.
  2. Error Propagation: Inaccurate correspondences from the first step can lead to errors that are amplified throughout the image generation process, resulting in low-fidelity outputs. Existing methods often lack reliable "prior guidance" to correct these errors.

Methods

PROMOTE tackles these challenges with two core innovations:

  1. Global-Local Contrastive Learning in Hyperbolic Space:
    • Global Contrastive Learning: Instead of using standard Euclidean distances, PROMOTE maps image features into a hyperbolic space. Hyperbolic geometry is better suited for capturing hierarchical and non-linear relationships, leading to more discriminative and domain-invariant features. A global contrastive loss is used to align these hyperbolic representations.
  2. Patch-Level Contrastive Learning: To further refine feature alignment, a local contrastive loss is introduced. This loss focuses on aligning the "semantic correlation distributions" of patches within the images, using the Fisher-Rao metric in hyperbolic space. This ensures that not only the overall image content but also the relationships between local features are consistent across domains.
  3. Prior-Guided Diffusion Model:
    • Self-Supervised Training: The model is trained in a self-supervised manner, meaning it generates its own ground truth data through data augmentation. This avoids the need for explicit paired datasets, which are often unavailable for image translation tasks.
  4. Structural Prior Guidance: The diffusion model, which generates images by iteratively denoising noise, is guided by the structural information from the input image (e.g., the sketch or pose). This "prior" is incorporated into the diffusion process by modifying the transition probabilities at each step. A novel "prior denoising loss" is derived to ensure that the generated image preserves the structure of the input while adopting the style of the exemplar.

Datasets

The paper evaluates PROMOTE on seven diverse datasets:

  • CelebA-HQ: High-quality celebrity faces.
  • Metfaces: Artistic facial avatars.
  • Meticulous: Chinese ink paintings.
  • Ukiyo-e: High-quality Ukiyo-e faces.
  • AAHQ: Facial avatars.
  • DeepFashion: Fashion images.
  • ADE-20k: A challenging dataset with segmentation masks, often used for scene parsing and image generation tasks.

Key Insights and Ideas

  • Hyperbolic Geometry for Image Translation: The paper demonstrates that hyperbolic geometry is a powerful tool for learning domain-invariant features in image translation, outperforming traditional Euclidean approaches. This is because hyperbolic spaces can better represent complex, hierarchical relationships present in image data.
  • Patch-Level Semantic Alignment is Crucial: Simply aligning global image features isn't enough. Aligning the relationships between local image patches (semantic correlation distributions) is key to achieving fine-grained control and high-fidelity translation.
  • Structural Priors Enhance Diffusion Models: Incorporating structural information from the input image as a "prior" into diffusion models significantly improves their ability to generate realistic images that maintain the desired structure. This is a novel way to guide the generative process beyond just denoising.
  • Self-Supervised Learning for Image Translation: The self-supervised training strategy is effective for exemplar-based tasks where paired data is scarce, allowing the model to learn robust mappings without explicit ground truth.

Key Results

  • Quantitative Improvements: PROMOTE achieves state-of-the-art performance across most of the seven tested datasets, significantly outperforming existing methods in terms of FID (Fréchet Inception Distance) and SWD (Sliced Wasserstein Distance), which measure image quality and realism. For example, on CelebA-HQ, PROMOTE achieved an FID of 11.0 and SWD of 12.0, outperforming previous bests.
  • Qualitative Improvements: Visual results show that PROMOTE generates images with higher fidelity, better style transfer, and more accurate semantic consistency compared to other methods. It also demonstrates an ability to generate local attributes (like glasses or earrings) with appropriate style, even if they are not explicitly present in the exemplar.
  • Ablation Studies: Removing components like global-local contrastive learning or the prior-guided diffusion model leads to a noticeable degradation in performance, validating the effectiveness of each proposed module.

Limitations and Future Work

  • Limitations: The paper acknowledges that while hyperbolic geometry offers advantages, it can be computationally more demanding than Euclidean space. The choice of curvature parameter in hyperbolic space might also require careful tuning.
  • Future Work: The authors suggest that their proposed framework, particularly the global-local contrastive learning and prior-guided diffusion, can be generalized to other generative tasks beyond exemplar-based image translation, such as contrastive representation learning and standard diffusion model applications. They also hint at further exploration of how to best leverage structural priors for enhanced generative capabilities.

Summary updated

Abstract

Exemplar-based image translation has garnered significant interest from researchers due to its broad applications in multimedia/multimodal processing. Existing methods primarily employ Euclidean-based losses to implicitly establish cross-domain correspondences between exemplar and conditional images, aiming to produce high-fidelity images. However, these methods often suffer from two challenges: 1) Insufficient excavation of domain-invariant features leads to low-quality cross-domain correspondences, and 2) Inaccurate correspondences result in errors propagated during the translation process due to a lack of reliable prior guidance. To tackle these issues, we propose a novel prior-guided diffusion model with global-local contrastive learning (PROMOTE), which is trained in a self-supervised manner. Technically, global-local contrastive learning is designed to align two cross-domain images within hyperbolic space and reduce the gap between their semantic correlation distributions using the Fisher-Rao metric, allowing the visual encoders to extract domain-invariant features more effectively. Moreover, a prior-guided diffusion model is developed that propagates the structural prior to all timesteps in the diffusion process. It is optimized by a novel prior denoising loss, mathematically derived from the transitions modified by prior information in a self-supervised manner, successfully alleviating the impact of inaccurate correspondences on image translation. Extensive experiments conducted across seven datasets demonstrate that our proposed PROMOTE significantly exceeds state-of-the-art performance in diverse exemplar-based image translation tasks.
BibTeX
@inproceedings{zhong2024promote,
  author = {Guojin Zhong and Yihu Guo and Jin Yuan and Qianjun Zhang and Weili Guan and Long Chen},
  title = {{PROMOTE: Prior-Guided Diffusion Model with Global-Local Contrastive Learning for Exemplar-Based Image Translation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680928},
}

Loading related content...

Figure