Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models

Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, Vishal M. PatelarXiv 2022 · OpenReview

TL;DR

This paper introduces a novel method for generating images based on multiple, diverse input conditions (modalities) using denoising diffusion probabilistic models. Their key innovation is a way to combine pre-trained diffusion models for individual modalities, avoiding the need for large, multi-modal paired datasets and enabling flexible generation with new combinations of conditions.

What Problem Does This Paper Address?

Generating realistic images that adhere to multiple, specific constraints (like a sketch, a text description, or a segmentation mask) is a challenging task. The primary hurdle is the scarcity of large datasets where every image is perfectly paired with all the desired conditional information. Existing methods often struggle with complex image distributions or require retraining the entire model when new conditions are introduced, making them inflexible and difficult to scale. This paper aims to overcome these limitations by enabling multi-modal image synthesis without requiring extensive multi-modal paired data.

Methods

The core of the proposed method, termed Multimodal Conditioning using Diffusion models (MCDM), lies in leveraging existing, pre-trained denoising diffusion probabilistic models (DDPMs) that are trained on individual modalities. Instead of retraining a single model on all modalities at once, MCDM combines the "scores" (gradients of the data distribution) from these individual models during the inference (generation) phase.

The authors derive a closed-form expression to compute an "effective score" that represents the combined information from multiple modalities. This effective score is then used within the standard DDPM sampling process to guide the generation towards an image that satisfies all the provided conditions. A key advantage is that this approach doesn't require retraining the individual models when new modalities are added or when combining existing ones in novel ways.

Datasets

The experiments were conducted on the CelebA dataset. The modalities used included:

  • Edge maps of images
  • Partial face segmentation masks (from CelebAHQ dataset)
  • Text captions

Key Insights and Ideas

  • Leveraging Pre-trained Models: The ability to combine pre-trained individual-modality diffusion models at inference time is a significant breakthrough. This circumvents the need for massive, multi-modal paired datasets, which are notoriously difficult to collect.
  • Scalability and Flexibility: The proposed method is inherently scalable. Adding a new modality or a new combination of modalities doesn't necessitate retraining the entire generative model, making it highly adaptable.
  • Closed-Form Solution: Deriving a closed-form expression for the effective score simplifies the combination process and makes it computationally tractable.
  • Diffusion Models for Multi-modal Synthesis: This work demonstrates the power and flexibility of diffusion models beyond single-condition generation, extending their utility to complex, multi-constraint scenarios.

Key Results

The paper presents qualitative results demonstrating the effectiveness of the MCDM approach across various combinations of modalities:

  • Sketch and Segment: The method successfully extracts complementary information from sketches and segmentation masks to fill in missing regions and generate coherent images.
  • Text and Segment: Text prompts were used to incorporate key details like hair color, gender, and mouth orientation into generated images, especially when segmentation masks had missing information.
  • Sketch and Text: Similar to the above, text prompts were used to influence specific attributes, such as hair color, in conjunction with sketch information.
  • Three Modalities (Sketch, Segment, Text): Experiments combining all three modalities showed successful generation guided by all inputs, with a specific example showing consistent "black hair" generation across multiple outputs.

The paper emphasizes that this is a "raw version" and more results and experiments are planned, suggesting these initial findings are promising proof-of-concept.

Limitations and Future Work

  • "Work in Progress": The authors explicitly state this is a "raw version" and a "work in progress," indicating that the presented results are preliminary.
  • Quantitative Evaluation: While qualitative results are shown, the paper does not detail extensive quantitative evaluations (e.g., FID scores, user studies) to rigorously benchmark performance against other methods.
  • Assumption of Independence: The derivation relies on the assumption that individual modalities are statistically independent. While this simplifies the math, it might not hold true in all real-world scenarios and could impact performance.
  • Future Work: The paper suggests that their method opens up possibilities for utilizing annotated data across multiple datasets for better-constrained generation. Further experiments and exploration of different modality combinations are implied.

Summary updated

Abstract

Image synthesis under multi-modal priors is a useful and challenging task that has received increasing attention in recent years. A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities (i.e. priors) and corresponding outputs. In recent work, a variational auto-encoder (VAE) model was trained in a weakly supervised manner to address this challenge. Since the generative power of VAEs is usually limited, it is difficult for this method to synthesize images belonging to complex distributions. To this end, we propose a solution based on a denoising diffusion probabilistic models to synthesise images under multi-model priors. Based on the fact that the distribution over each time step in the diffusion model is Gaussian, in this work we show that there exists a closed-form expression to the generate the image corresponds to the given modalities. The proposed solution does not require explicit retraining for all modalities and can leverage the outputs of individual modalities to generate realistic images according to different constraints. We conduct studies on two real-world datasets to demonstrate the effectiveness of our approach
BibTeX
@misc{nair2022image,
  author = {Nithin Gopalakrishnan Nair and Wele Gedara Chaminda Bandara and Vishal M. Patel},
  title = {{Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models}},
  year = {2022},
  eprint = {2206.05039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.05039},
}
Figure