Diffusion models are generators that learn to turn random noise into samples by repeatedly denoising. They have become strong priors for inverse problems such as deblurring, super-resolution, inpainting and CT reconstruction, where one image has to be recovered from a corrupted measurement. Their weakness is cost. Solvers like diffusion posterior sampling run hundreds of denoiser calls per image, and at each call they backpropagate through the network to pull the estimate toward the measurement. That makes every reconstruction a small test-time optimization: it needs backward passes, stored activations, and seconds of GPU time. Every image pays the same price, whether it is a mild 2× upscale of a smooth face or a heavily masked cluttered bedroom.
This note proposes that the denoiser itself can tell us how much refinement each image needs. A denoiser trained with squared error computes the posterior mean of the clean image. By Tweedie's formula, it therefore also encodes the posterior's local spread. From these two quantities we build forward-only tests. The first decides how noisy an initial guess effectively is, which sets the step at which refinement should start. The second decides when further steps can no longer move the answer by more than its own uncertainty, which sets when to stop. The claim is not a new sampler. It is a per-image compute-allocation rule that wraps existing training-free solvers. It should be judged by one thing: whether it beats the best fixed short schedule at the same average compute.
Core Claim
A pre-trained denoiser $D_\sigma$ at noise level $\sigma$ satisfies Tweedie's identity: $D_\sigma(x)=\mathbb{E}[x_0\mid x]=x+\sigma^2\nabla_x\log p_\sigma(x)$, where $x_0$ is the clean image and $x=x_0+\sigma\epsilon$ is its noisy version. This identity is the basis of the denoiser-prior solvers in Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser and Bayesian Imaging Using Plug & Play Priors: When Langevin Meets Tweedie. The second-order version, which Tweedie Moment Projected Diffusions for Inverse Problems uses to precondition guidance, gives the posterior covariance through the denoiser's Jacobian. Its trace is a single number, the total posterior variance:
$$ v_\sigma(x)=\operatorname{tr}\,\mathrm{Cov}[x_0\mid x]=\sigma^2\,\operatorname{tr}\,\nabla_x D_\sigma(x). $$
We estimate $v_\sigma$ with a few random probes. Each probe uses a Jacobian-vector product, or a finite difference $D_\sigma(x+\delta u)-D_\sigma(x)$, and needs no backward pass. Let $\hat{x}_0^{(k)}$ be the solver's posterior-mean estimate after step $k$, let $H$ be the measurement operator, let $y$ be the measurement, $\sigma_y$ its noise level and $m$ its dimension. We stop at the first step $k$ where
$$ \|\hat{x}_0^{(k)}-\hat{x}_0^{(k-1)}\|^2<\tau\, v_{\sigma_k}(x_k)\quad\text{and}\quad \|y-H\hat{x}_0^{(k)}\|^2\le (1+\gamma)\,m\,\sigma_y^2 . $$
In words, we stop once the estimate moves less than a fraction $\tau$ of its own posterior uncertainty and agrees with the measurement to within the noise level (a margin $\gamma$ allows for slack). The second condition is the classical discrepancy principle. The first is the new ingredient. The entry point is chosen in a similar spirit. We re-noise a cheap initial reconstruction (a pseudo-inverse, or a fast task network) and compare the denoiser's residual energy $\|D_\sigma(\tilde{x})-\tilde{x}\|^2$ against a curve calibrated offline on clean images only. We then start at the smallest $\sigma$ whose residual looks like that of genuine noisy clean data.
The assumption behind the proposal is that the steps an image truly needs vary a lot from image to image. Where that holds, a fixed budget is wasted on easy inputs and too small for hard ones, and a per-image rule recovers the difference.
Relation to Prior Work
Three lines of work motivate the proposal and supply its parts.
The first shows that single, well-chosen evaluations of the score carry a lot of information. DUSA replaces hundreds of Monte Carlo diffusion steps with one evaluation at an intermediate timestep ($t=100$) to guide classifier adaptation. We borrow the idea that one timestep can be informative, but we apply it to deciding how much generative refinement to run, rather than to adapting a classifier by gradient descent.
The second line attacks refinement latency with fixed designs. DGSolver reduces discretization error through higher-order ODE solvers and caching. ROPA solves all timesteps in parallel and adapts its Jacobian bandwidth within one trajectory, but it targets unconditional and text-to-image generation and helps little at 1–4 steps. CoSIGN reaches 1–2 steps by training a ControlNet for each degradation. APS distills the posterior into a one-pass flow. All four fix the compute per image in advance, or buy speed with offline training. Ours adapts compute per input and needs no training that depends on the degradation.
The third line shapes our design by showing what fails at test time. SNAP and T3A show that avoiding backward passes, through normalization or prototype updates, is what makes adaptation fast. Test-Time Training with Self-Supervision and Self-supervised Test-time Adaptation on Video Data show that per-sample backpropagation is the dominant cost. This is why our certificates are forward-only. It is also why we count backward passes, not just denoiser calls.
One result needs to be addressed directly. Diffusion Posterior Sampling is Computationally Intractable proves that posterior sampling can require superpolynomial time in the worst case. Our stopping test therefore cannot certify that samples are correct. It certifies only that more local refinement would change the estimate less than its own Tweedie uncertainty. For posteriors that break into many isolated modes, this test can stop too early inside one mode. We treat that as a failure mode to measure, not one to explain away.
Research Program
The program has three parts.
Measurement study. Before building anything, we measure how heterogeneous the need for compute really is. For DPS and a projection-based solver on FFHQ 256×256 and LSUN Bedrooms, covering super-resolution, Gaussian and motion deblurring, and box and random inpainting, we sweep start step and step count per image. For each image we record an oracle: the smallest budget within 0.1 dB PSNR, or 0.005 LPIPS, of its best result. The spread of these oracle budgets bounds what any adaptive rule can gain.
Certificates and entry rule. We build the two certificates and the entry rule. We study how many variance probes are needed, whether finite differences are accurate enough to replace Jacobian-vector products, and how the offline residual-energy curve transfers across datasets. The thresholds $\tau$ and $\gamma$ are set once on a small validation split and then frozen across tasks.
Backward passes as a separate budget. We treat the number of backward passes as its own budget. Once the certificate says an image is close to done, the rule switches from backpropagated guidance to forward-only projection steps on the measurement (closed-form for linear $H$, as in CoSIGN's hard constraint). We test whether this cuts memory and latency beyond what cutting denoiser calls alone achieves.
Baselines
The naive reference is full-length DPS with its standard schedule. APS reports 27.93 dB PSNR on CelebA denoising for DPS, at about 16–19 seconds per image. We rerun it with the same pretrained denoiser, hardware and measurement noise as every other method, so that all latencies are comparable.
The strongest competitor on the question we are asking is the best tuned fixed short schedule: the same solver with one global start step and step count, grid-searched on the same validation split with the same number of tuning trials as our thresholds. This is the honest opponent, because our only claim is that allocating per image beats allocating globally. We win only if the adaptive rule sits on a strictly better quality–latency Pareto curve at matched mean compute. Concretely, that means at least 0.3 dB PSNR, or a clear LPIPS/FID margin, at equal mean denoiser calls. Alternatively, it means equal quality at 25% lower mean wall-clock time.
Among fast methods that do use training, CoSIGN is the strongest point of comparison. It reports FID 38.64 on LSUN Bedrooms against 43.54 for 1000-step DPS, at under 100 ms per image. On AAPM low-dose CT it reports 34.26 dB against 25.52 dB for DPS. APS reports 25.82 dB on CelebA at 0.002 s. We do not expect to match their latency. Instead we compare on held-out degradations those methods were not trained for, where their fixed networks should degrade.
Three ablations isolate the key ingredient. The first stops on the discrepancy condition alone, with no Tweedie variance. The second uses a fixed entry step with adaptive stopping. The third uses the variance certificate, but with the Tweedie trace replaced by the spread across two independent runs of the solver, a cheaper uncertainty proxy.
Why not just…?
Why not just tune a shorter fixed schedule? This is the real threat. If the measurement study finds that the oracle budgets cluster tightly, a single well-tuned short schedule captures nearly everything and the proposal should be dropped. The oracle-spread measurement in the first part of the program decides this before any method work begins. The Pareto comparison against the tuned fixed schedule decides it again at the end.
Why not just distill into a one-step model? Distillation is the right answer when the degradation family is known in advance. APS, however, loses about 2 dB against DPS on CelebA and shows high-frequency graininess. CoSIGN needs a trained encoder for each degradation. Our held-out-degradation experiments show whether an adaptive training-free solver keeps its quality where the distilled models do not. On in-distribution tasks we expect to lose on latency, and we will report that loss.
Why not just stop when the measurement fits (discrepancy principle)? Under heavy inpainting or compressive measurements, many images fit $y$ equally well while the unobserved regions are still changing. A residual-only rule should therefore stop too early exactly on the hard cases. The residual-only ablation tests this directly, with the effect measured in masked regions separately from observed ones.
Why not just train a small network to predict the needed step count? Such a predictor would need paired degraded and clean data for each task, which gives up the training-free property. It would also inherit the same problems under distribution shift that SNAP and T3A address for classifiers. We include a regressor trained on the oracle budgets as an upper-bound comparison on in-distribution tasks only.
Proposed Experiments
We evaluate on FFHQ and CelebA faces, LSUN Bedrooms, and AAPM LDCT sparse-view CT with 23 angles. Metrics are PSNR, LPIPS and FID. For cost we report mean and 95th-percentile wall-clock time, denoiser calls, backward passes, and peak memory.
There are three decisive checks. First, the oracle study must show a large spread in needed budget per image. If most images need within ±20% of the median, the idea fails. Second, at matched mean compute, the adaptive rule must beat the tuned fixed schedule on at least three of the five tasks. Third, the variance-based stop must beat the residual-only stop on masked-region error.
We also run a stress test against the intractability result. On a Gaussian-mixture toy posterior, of the kind TMPD uses to expose mode collapse, we measure how often the certificate stops inside one mode while Wasserstein error to the true posterior is still high. This bounds when the rule should not be trusted. Finally, we test transfer to a degradation held out from all tuning, motion blur whose kernels were never seen, against CoSIGN and APS models trained on Gaussian blur.
References
- Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
- Bayesian Imaging Using Plug & Play Priors: When Langevin Meets Tweedie
- Tweedie Moment Projected Diffusions for Inverse Problems
- Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation
- DGSolver: Diffusion Generalist Solver with Universal Posterior Sampling for Image Restoration
- Robust Parallel Diffusion Sampling via Dynamic Jacobian Bandwidth
- CoSIGN: Few-Step Guidance of ConSIstency Model to Solve General INverse Problems
- Amortized Posterior Sampling with Diffusion Prior Distillation
- SNAP: Low-Latency Test-Time Adaptation with Sparse Updates
- Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization
- Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
- Self-supervised Test-time Adaptation on Video Data
- Diffusion Posterior Sampling is Computationally Intractable