Standard diffusion models generate data by starting from a blank canvas of pure Gaussian noise and iteratively removing that noise over hundreds of sampling steps. While this "noise-to-data" paradigm works exceptionally well for unconditional generation, it introduces a glaring computational redundancy when applied to conditional tasks like image restoration, speech enhancement, or medical image segmentation.
In these tasks, the input signal—whether it is a blurry photograph, a cloudy satellite image, or a noisy audio track—is not completely devoid of information. It already contains the vast majority of the target's structural layout, coarse geometry, and global context. Forcing a model to first destroy this existing structure by mapping it to pure noise, only to painstakingly reconstruct it from scratch, is highly inefficient.
A rapidly growing body of research addresses this bottleneck by bypassing pure noise initialization entirely. Instead of starting from scratch, these methods establish direct "data-to-data" trajectories, utilize mean-reverting stochastic processes, or truncate the diffusion chain to start from informative, structurally sound priors.
The Redundancy of Reconstructing the Skeleton
To understand why starting from pure noise is inefficient, we have to look at how diffusion models process information across different timesteps. During the reverse (denoising) process, a diffusion model typically resolves features from coarse to fine:
- Early stages establish global layout, coarse shapes, and low-frequency structures.
- Late stages fill in high-frequency details, fine textures, and sharp edges.
When performing conditional generation, the low-quality input already provides the global "skeleton" of the scene. As Bin Xia et al. observe in DiffIR: Efficient Diffusion Model for Image Restoration, estimating every pixel or feature map from scratch completely ignores these existing structural constraints, leading to massive and unnecessary computational expenditure.
Furthermore, injecting uninformative low-frequency noise into an already structured image can actively harm performance. As explained by Zhen Wang et al. in NFCDS: A Plug-and-Play Noise Frequency-Controlled Diffusion Sampling Strategy for Image Restoration, low-frequency noise introduces random structural perturbations that the model must waste steps trying to correct. By recognizing that the initial stages of the reverse process redundantly reconstruct information that is already available, authors like Yu-Shan Tai et al. demonstrate that systems can safely bypass the earliest denoising steps without losing the coarse features or layout of the original scene.
Mechanism 1: Direct Data-to-Data Schrödinger Bridges
Instead of routing the generative path through an uninformative Gaussian distribution, how can we map a direct path from the degraded source distribution to the clean target distribution?
The Schrödinger Bridge (SB) framework provides a mathematically rigorous solution to this problem. Rather than defining a forward process that diffuses data into pure noise, a Schrödinger Bridge solves for the most likely non-linear probability trajectory between two arbitrary distributions.
Standard Diffusion: Degraded Input ──(Conditioning)──> [ Pure Noise ] ──(Denoise)──> Clean Output
Schrödinger Bridge: Degraded Input ─────────────────────────────────────────────────> Clean Output
In I2SB: Image-to-Image Schrödinger Bridge, Guan-Horng Liu et al. break away from the standard Gaussian requirement, learning a direct non-linear path between degraded and clean image distributions. This formulation allows the model to be highly robust to low-step sampling because it never has to guess the global structure from pure randomness.
This paradigm shift from "noise-to-data" to "data-to-data" is highly versatile:
- Conditional Simulation: Yuyang Shi et al. leverage this in Conditional Simulation Using Diffusion Schrödinger Bridges to incorporate informative priors directly into the bridge endpoints.
- Audio Synthesis: In BridgeVoC: Neural Vocoder with Schrödinger Bridge, Tong Lei et al. treat vocoding as a direct spectrogram restoration task, bypassing the need to reconstruct basic acoustic features from pure noise.
Mechanism 2: Mean-Reverting SDEs as Structural Anchors
Another approach modifies the underlying Stochastic Differential Equations (SDEs) so that the noise process is inherently anchored to the degraded input.
In standard diffusion, the drift term of the SDE pulls the data distribution toward a zero-mean Gaussian. In a mean-reverting SDE, the drift term is modified to pull the distribution back toward a specific target mean—in this case, the degraded input image itself.
In Image Restoration with Mean-Reverting Stochastic Differential Equations, Ziwei Luo et al. design a process where the degradation path leads from a high-quality image directly to its blurry or rainy counterpart, rather than to pure noise. Because the process is anchored to the structured degradation, the training objective becomes significantly more stable.
This mean-reverting behavior acts as a structural safeguard. For instance, Yi Liu et al. demonstrate in Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space that treating a cloudy image as a structural anchor prevents the generative hallucinations and pixel-level inconsistencies common to standard noise-initialized models. To further streamline this, Ziwei Luo et al. introduced Forward-only Diffusion Probabilistic Models, which eliminates the traditional coupled forward-backward scheme entirely, evolving the source distribution into the target through a single, analytically tractable process.
Mechanism 3: Shortcut Sampling and Informative Priors
If the early stages of the diffusion process are computationally redundant because they merely recreate coarse structures, a straightforward optimization is to skip them entirely. This strategy is known as shortcut or truncated sampling.
Instead of initializing the reverse process at maximum variance ($t = T$), these methods identify an intermediate "transitional state" ($t = t_{start}$) where the noise level matches the residual uncertainty of the task.
Standard Chain: [t=T: Pure Noise] ──> ... ──> [t=t_start: Coarse Layout] ──> ... ──> [t=0: Clean Data]
Shortcut Chain: [t=t_start: Rough Draft] ──> ... ──> [t=0: Clean Data]
▲
(Initialized from Prior)
Several distinct strategies have emerged to initialize these shortcuts:
- Transitional Bridging States: In Accelerating Diffusion Models via Inverse Problems through Shortcut Sampling, Gongye Liu et al. map out a transitional state that bridges the gap between low-quality inputs and high-quality targets, achieving high-fidelity restoration in a fraction of the steps.
- Neural Network "Rough Drafts": In Accelerating Diffusion Models via Pre-segmentation Diffusion Sampling for Medical Image Segmentation, Xutao Guo et al. initialize the denoising process from a noisy approximation provided by a standard, fast neural network. The diffusion model is then freed from the burden of global layout generation and can focus entirely on refining boundaries and correcting specific errors.
- Prior-Aware Initialization: For domain-specific translations like bokeh rendering, Shiyue Yan demonstrates in Reschedule Diffusion-based Bokeh Rendering that a prior-aware initialization based directly on the input image eliminates common artifacts like color drift and structural inconsistency.
- Jointly Learned Priors: Rather than relying on heuristics, Ching Hua Lee et al. propose RestoreGrad: Signal Restoration Using Conditional Denoising Diffusion Models with Jointly Learned Prior, which integrates a VAE-like framework to jointly learn an informative starting prior, accelerating both training convergence and inference.
By restricting the diffusion process to a narrow window of refinement rather than a full reconstruction from scratch, these methods preserve the structural integrity of the source data while drastically reducing the number of required function evaluations.
Key Foundational Work
I2SB: Image-to-Image Schrödinger Bridge (2023) — contributed 2 concept elements to this direction.
Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction (2022) — contributed 1 concept elements to this direction.
Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancement (2023) — contributed 1 concept elements to this direction.