Reschedule Diffusion-based Bokeh Rendering

Shiyue Yan, Xiaoshi Qiu, Qingmin Liao, Jing-Hao Xue, Shaojun Liu

Abstract

Bokeh rendering for images shot with small apertures has drawn much attention in practice. Very recently people start to explore diffusion models for bokeh rendering, aiming to leverage the models' surging power of image generation. However, we can clearly observe two big issues with the images rendered by diffusion models: large fluctuation and severe color deviation. To address these issues, we propose in this paper a prior-aware sampling approach, which can adaptively control the noise scale through learned priors, and a prior-aware noise scheduling strategy, which can greatly reduce the number of inference steps without sacrificing performance. Extensive experiments show that our method can effectively alleviate the fluctuation problem of sampling results while ensuring similar color styles to the input image. In addition, our method outperforms state-of-the-art methods, sometimes even with only two steps of sampling. Our code is available at https://github.com/Loeiii/Reschedule-Diffusion-based-Bokeh-Rendering.

Introduction

Digital bokeh rendering—the simulation of background blur typically achieved by high-end cameras with large apertures—is a staple of modern computational photography. While diffusion models have revolutionized image generation, their application to bokeh rendering has been hindered by three critical challenges: output fluctuation, severe color deviation, and high computational latency. Standard diffusion processes start from pure noise, which often leads to inconsistent results and color shifts compared to the original sharp input image. Furthermore, the iterative nature of these models typically requires dozens of steps, making them impractical for real-time mobile use.

Methodology: Prior-Aware Diffusion

To overcome these limitations, we introduce Reschedule Diffusion, which "tames" the generative process by leveraging the information already present in the sharp input image. Our approach consists of two primary innovations.

Prior-aware Sampling

Instead of starting from pure Gaussian noise, we initialize the process with a noisy version of the input image itself. By applying an adaptive noise scale based on the $3\sigma$ rule of statistics, we anchor the model to the original scene's properties. This "prior-aware" initialization prevents the model from drifting away from the original colors and structures, effectively solving the color deviation problem.

Analysis of the relationship between Mean Squared Error (MSE) and Noise-to-Signal Ratio (NSR), alongside a visual representation of how noise distributions merge as NSR increases.

Prior-aware Noise Scheduling

We redesigned the noise schedule to prioritize the preservation of the input "prior." This rescheduling allows the model to produce high-quality results in as few as 2 to 10 steps. This represents a massive leap in efficiency, enabling high-quality bokeh rendering with minimal computational cost.

Experimental Evaluation

The proposed method was evaluated on the FPBNet dataset, a benchmark containing 941 high-resolution (2232×1488) aligned image pairs of small-aperture and large-aperture shots.

Qualitative Comparison

As shown in the comparison below, our method effectively eliminates the color shifts and structural inconsistencies seen in standard diffusion baselines. The rendered bokeh is both aesthetically pleasing and faithful to the original image's color palette and lighting.

Qualitative comparison of bokeh rendering results: (a) input images, (b) results from standard diffusion models showing noticeable color deviation and artifacts, and (c) results from our proposed Reschedule Diffusion method.

Quantitative Performance

  • Efficiency: Our model achieves a PSNR of 30.99 with only 5 steps, outperforming the baseline which achieved a lower score of 30.25 with significantly more computation.
  • Perceptual Quality: We achieved state-of-the-art scores in LPIPS, surpassing existing methods like BRViT and Refusion.
  • Consistency: The prior-aware initialization reduces output fluctuation to levels comparable to deterministic, non-generative models.

Conclusion and Future Work

Reschedule Diffusion provides a fast, stable, and color-accurate solution for bokeh rendering. By replacing random noise with a prior-aware starting point, we solve the core issues of generative bokeh rendering. Future work will focus on generalizing this approach to other image restoration tasks and optimizing performance for deployment on mobile hardware.

Summary updated

Abstract

Bokeh rendering for images shot with small apertures has drawn much attention in practice. Very recently people start to explore diffusion models for bokeh rendering, aiming to leverage the models' surging power of image generation. However, we can clearly observe two big issues with the images rendered by diffusion models: large fluctuation and severe color deviation. To address these issues, we propose in this paper a prior-aware sampling approach, which can adaptively control the noise scale through learned priors, and a prior-aware noise scheduling strategy, which can greatly reduce the number of inference steps without sacrificing performance. Extensive experiments show that our method can effectively alleviate the fluctuation problem of sampling results while ensuring similar color styles to the input image. In addition, our method outperforms state-of-the-art methods, sometimes even with only two steps of sampling. Our code is available at https://github.com/Loeiii/Reschedule-Diffusion-based-Bokeh-Rendering.
BibTeX
@inproceedings{yan2024reschedule,
  author = {Shiyue Yan and Xiaoshi Qiu and Qingmin Liao and Jing-Hao Xue and Shaojun Liu},
  title = {{Reschedule Diffusion-based Bokeh Rendering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=eX1V6yGyYS},
}
Figure