Reschedule Diffusion-based Bokeh Rendering

Shiyue Yan, Xiaoshi Qiu, Qingmin Liao, Jing‐Hao Xue, Shaojun Liu, Weisheng Dong, Xin Li, Guangming Shi

Abstract

Diffusion models (DMs) have revolutionized image synthesis but face significant challenges in bokeh rendering, specifically regarding stochastic output fluctuations and severe color deviations. We present a prior-aware diffusion framework that addresses these issues by initializing the reverse diffusion process with a noisy version of the original small-aperture image rather than pure Gaussian noise. By employing an adaptive prior magnitude based on the 3-sigma rule and a refined noise scheduling strategy, our method achieves state-of-the-art results in as few as two to five inference steps. This approach significantly reduces computational overhead while maintaining high perceptual quality and color consistency, outperforming existing benchmarks on the FPBNet dataset.

Introduction

Bokeh rendering—the simulation of a shallow depth of field—is a staple of modern computational photography. While Diffusion Models (DMs) are the current state-of-the-art in image generation, applying them to bokeh has been historically difficult. Standard DMs often produce inconsistent results across multiple runs and suffer from color shifts (e.g., unnatural blue or yellow tints) because they start from "cold" Gaussian noise, losing the context of the original photo. Furthermore, the high iteration counts required by standard diffusion are impractical for mobile applications. This paper introduces a framework that anchors the diffusion process to the original image's structure and color profile to solve these limitations.

Methodology

Our framework introduces two primary innovations to the standard Denoising Diffusion Probabilistic Model (DDPM) pipeline: Prior-aware Sampling and Prior-aware Noise Scheduling.

Prior-aware Sampling

Instead of starting the reverse process at $T=1000$ with pure noise, we initialize the process using the input small-aperture image ($y$) mixed with a specific amount of noise. We calculate an adaptive "prior magnitude" ($A_i$) based on the standard deviation of the input image's color channels. This ensures the model stays anchored to the original image's DNA.

Prior-aware Noise Scheduling

To prevent the model from over-relying on the input image at low step counts, we utilize a refined scheduling formula with a weighting factor $(1 - e^{-N/20})$. This balances the influence of the original image prior against the generative power of the diffusion model, even at extremely low step counts (N=2 to 5).

Prior-aware noise analysis showing the relationship between NSR and MSE.

As shown in Figure 4, the statistical 3-sigma rule is used to determine the minimal Noise-to-Signal Ratio (NSR). This allows the model to mathematically decide the exact amount of noise needed to make the input image compatible with the diffusion model's latent states without sacrificing color consistency.

Experimental Results

The proposed method was evaluated on the FPBNet dataset, which contains 941 sets of high-resolution, real-world image pairs.

Quantitative Performance

Our model, "Ours (N=5)", achieved an LPIPS score of 0.0678, a significant improvement over the previous state-of-the-art of 0.0834. Even at a hyper-efficient N=2 steps, the model achieved a PSNR of 30.54, outperforming many existing methods that require significantly more computation.

Qualitative Analysis

Qualitative results demonstrate that the method effectively eliminates the "cold/hot" color shifts seen in standard diffusion approaches. The model excels at keeping foreground edges sharp while rendering a natural, creamy blur in the background.

Qualitative comparison of bokeh rendering performance across different configurations.

Conclusion

By leveraging the mutual information between small-aperture inputs and large-aperture targets, our prior-aware diffusion framework provides a robust and efficient solution for bokeh rendering. The model's ability to achieve high-quality results in just 5 steps makes it a viable candidate for real-time mobile deployment. Future work will explore applying this prior-aware rescheduling to other restoration tasks like super-resolution and deblurring.

Summary updated

BibTeX
@inproceedings{yan2024reschedule,
  author = {Shiyue Yan and Xiaoshi Qiu and Qingmin Liao and Jing‐Hao Xue and Shaojun Liu and Weisheng Dong and Xin Li and Guangming Shi},
  title = {{Reschedule Diffusion-based Bokeh Rendering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=eX1V6yGyYS},
}
Figure