The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, Chongxuan Li

Abstract

We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.

Introduction

Image editing with diffusion models typically follows a two-stage pipeline: inverting an image into a latent noise representation and then sampling from that manipulated noise to generate a new image. Traditionally, deterministic Ordinary Differential Equations (ODEs), such as DDIM, have been the preferred choice due to their perfect invertibility. However, the act of editing introduces a "distribution mismatch"—the edited latent no longer aligns with the natural noise distribution the model was trained on. This paper identifies that ODEs, being invariant, preserve this mismatch, leading to artifacts. In contrast, Stochastic Differential Equations (SDEs) leverage randomness as a corrective force to pull edited images back toward the natural data manifold.

Theoretical Framework: SDE vs. ODE

The core contribution of this work is a unified probabilistic formulation for diffusion-based editing. By analyzing the Fokker-Planck equation, the authors prove a fundamental difference in how SDEs and ODEs handle distribution shifts during the reverse process.

Theoretical comparison between SDE and ODE frameworks, illustrating the contractive nature of SDEs and the SDE-Drag copy-and-paste mechanism.

As shown in the figure above, SDEs are "contractive." As the sampling time approaches zero, the Kullback-Leibler (KL) divergence between the marginal distribution of the edited SDE and the original data distribution decreases. In simpler terms, the inherent noise in SDEs helps "heal" the image by forgetting the unnatural artifacts of the edit. ODEs, conversely, are "invariant," meaning they faithfully carry the editing errors through to the final output.

Proposed Methods

Cycle-SDE

To address the challenge of inverting stochastic processes, the authors employ Cycle-SDE. This method logs the specific noise added during the forward (inversion) pass and re-injects it during the reverse pass. This maintains structural consistency while allowing the model to benefit from the stochastic refinement that improves image quality.

SDE-Drag

Building on the contractive property of SDEs, the authors propose SDE-Drag, a point-based content dragging method. Unlike previous approaches like DragGAN that rely on complex latent space optimization, SDE-Drag uses a remarkably simple "copy-and-paste" operation. Noisy pixels from the source region are moved to the target location in the latent space. The SDE's natural tendency to pull samples toward the data distribution then seamlessly integrates these moved pixels into the new context.

Experimental Results

The researchers evaluated SDE-based editing across inpainting, image-to-image translation, and a new benchmark called DragBench.

Visual results of SDE-Drag across various categories including animals, landscapes, and AI-generated art.

DragBench Performance

DragBench consists of 100 high-quality image-caption pairs across five categories. In user studies, SDE-Drag was preferred over 80% of the time compared to ODE baselines. It also significantly outperformed DragGAN, demonstrating the power of a mathematically grounded approach over heavy optimization.

Quantitative Improvements

  • Inpainting: Inpaint-SDE achieved an FID of 4.91 compared to 5.86 for Inpaint-ODE on the Places dataset.
  • Consistency: SDE methods showed superior alignment with text prompts, as measured by CLIP-scores, indicating that the stochastic process better respects the semantic guidance of the model.

Conclusion

This work demonstrates that randomness is not a hindrance but a "blessing" in diffusion-based image editing. By providing a theoretical foundation for the superiority of SDEs over ODEs, the authors pave the way for more robust and efficient editing tools. While challenges remain regarding numerical stability at high guidance scales and complex identity preservation, SDE-Drag represents a significant step forward in making point-based image manipulation both simpler and more effective.

Summary updated

Abstract

We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.
BibTeX
@inproceedings{nie2024blessing,
  author = {Shen Nie and Hanzhong Allan Guo and Cheng Lu and Yuhao Zhou and Chenyu Zheng and Chongxuan Li},
  title = {{The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=DesYwmUG00},
}
Figure