The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, Chongxuan Li

In the world of diffusion models, speed and determinism have long made Ordinary Differential Equations (ODEs)—like the popular DDIM sampler—the go-to choice for image editing. However, a new paper presented at ICLR 2024, "The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing," argues that we’ve been overlooking a fundamental advantage of Stochastic Differential Equations (SDEs). By embracing randomness, we can achieve significantly higher fidelity and better alignment with our editing goals.

TL;DR

This paper proves that SDEs are theoretically superior to ODEs for image editing because the added noise acts as a corrective force, pulling edited images back toward the true data distribution. The authors leverage this "blessing of randomness" to create SDE-Drag, a simple yet powerful point-based editing tool that consistently outperforms existing state-of-the-art methods.

What Problem Does This Paper Address?

Diffusion-based image editing typically involves two stages: first, converting an image into a "latent" representation (often by adding noise or inverting the diffusion process), and second, manipulating that latent to achieve a desired change (like dragging an object or changing a text prompt).

The problem is that any manual manipulation of the latent variable inevitably pushes it away from the "natural" distribution the model was trained on. This is called distribution mismatch. Most current methods use ODE samplers because they are fast and easy to reverse. However, the authors identify a critical flaw: ODEs are "invariant," meaning they preserve the errors introduced during the editing stage all the way to the final image. This often results in artifacts, blurry textures, or edits that don't quite look "real."

Methods

The researchers tackle this by first establishing a unified probabilistic formulation for diffusion editing, covering tasks from inpainting to image-to-image translation.

Figure 1: Overview of the paper. (a) Technical contributions. (b) Visualization of SDE-Drag.

The Theoretical Proof

The core of the paper is a mathematical proof using Kullback-Leibler (KL) divergence. They demonstrate that:

  • SDEs are contractive: The gap between the edited distribution and the true data distribution gradually decreases as the sampling process moves toward the final image. The randomness "blesses" the process by correcting the initial manipulation errors.
  • ODEs are invariant: The distribution gap remains constant. If your initial edit was slightly "off-manifold," the ODE sampler will keep it there.

SDE-Drag

Building on this theory, they introduce SDE-Drag for point-based content dragging. Unlike previous methods that require complex, time-consuming optimization in the latent space, SDE-Drag uses a straightforward copy-and-paste approach on the noisy latents. It then employs Cycle-SDE—a clever way to use SDEs while maintaining the ability to reconstruct the original image's details—to generate the final result.

Datasets

The authors evaluated their approach across several standard tasks and introduced a new benchmark:

  • Places Dataset: Used for evaluating inpainting performance.
  • COCO & COCO-BISON: Used for image-to-image translation experiments.
  • DragBench: A new, challenging benchmark introduced in this paper. It consists of 100 image-caption pairs featuring natural scenes, art, and AI-generated images. This is a significant step up from previous benchmarks that were often limited to specific domains like human faces.

Key Insights and Ideas

  • Randomness as a Feature, Not a Bug: In generative modeling, noise is usually seen as something to be removed. Here, it is revealed as a vital regularizer that pulls "broken" edited latents back into the realm of plausible natural images.
  • Simplicity via Theory: By proving that SDEs naturally handle distribution mismatches, the authors were able to replace complex optimization loops (used in methods like DragGAN or DragDiffusion) with a simple copy-paste latent manipulation.
  • Cycle-SDE Utility: The paper highlights Cycle-SDE as a robust tool for maintaining image fidelity during stochastic editing, effectively bridging the gap between the flexibility of SDEs and the reconstruction precision of ODEs.

Key Results

The results across all tested tasks were consistent: SDE-based methods outperformed their ODE counterparts.

  • Inpainting: Inpaint-SDE achieved better FID and LPIPS scores than Inpaint-ODE across various mask sizes and step counts. Notably, SDE with only 25 steps outperformed ODE with 100 steps.
  • Dragging: In a user study on the new DragBench, SDE-Drag was preferred over 80% of the time compared to the ODE baseline. It also significantly outperformed specialized models like DragGAN and DragDiffusion.
  • Efficiency: Despite the theoretical complexity, SDE-Drag is highly efficient. It takes roughly the same amount of time per image as direct ODE baselines (approx. 76 seconds on an NVIDIA A100).

Limitations and Future Work

While the results are impressive, the authors note a few areas for further exploration:

  • Theoretical Assumptions: The current proof assumes the model has perfectly learned the data's score function and ignores errors introduced by discretizing the SDE/ODE for numerical solvers.
  • Cycle-SDE Convergence: Mathematically proving the convergence rates for Cycle-SDE is challenging because the noise terms in the sampling path are correlated, unlike standard SDEs.
  • The "Open-Set" Frontier: While SDE-Drag handles diverse images better than previous tools, dragging complex objects in completely unconstrained "open-set" environments remains a difficult challenge with occasional failures.

Summary updated

Abstract

We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.
BibTeX
@inproceedings{nie2024blessing,
  author = {Shen Nie and Hanzhong Allan Guo and Cheng Lu and Yuhao Zhou and Chenyu Zheng and Chongxuan Li},
  title = {{The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=DesYwmUG00},
}
Figure