Diffusion models are generators that learn to turn random noise into images by denoising it step by step. They are now widely used to solve inverse problems such as deblurring, super-resolution, and inpainting without retraining: the solver alternates a generative denoising step with a "data-consistency" step that pulls the image back toward agreement with the measurement. To make these solvers fast, people add momentum, meaning they extrapolate from past iterates the way Nesterov or heavy-ball methods do. Optimization theory has a precise warning about this. Momentum speeds convergence, but it also amplifies whatever noise enters the iteration, and the amplification depends on how strongly the problem constrains each direction. Diffusion solvers inject noise on purpose, because that noise is where realistic texture comes from.
This note proposes treating the convergence-robustness trade-off of accelerated optimization as a dial on the perception-distortion trade-off. Distortion is pixel error against the ground truth; perception is how natural the output distribution looks. For linear measurement operators with a known singular value decomposition, we assign each singular mode its own momentum coefficient. Each coefficient is the fastest one whose predicted steady-state noise amplification stays under a target budget, and the prediction comes from the closed-form variance of a noisy two-step linear recursion. A single budget parameter then sweeps the solver from fast and faithful to fast and textured. It does this analytically, at fixed step count, and without the offline training that current learned extrapolation needs. The claim to be tested is that this per-mode allocation traces a better few-step perception-distortion frontier than uniform momentum, learned extrapolation, or spectral noise filtering.
Core Claim
Robustness of accelerated first-order algorithms for strongly convex optimization problems models gradient methods on quadratics as linear dynamical systems. It shows that the steady-state variance of noisy iterates depends on the whole Hessian spectrum, and that acceleration inflates it by roughly $\sqrt{\kappa}$, where $\kappa$ is the condition number. Read mode by mode, this has a direct image-restoration meaning. Directions the measurement constrains well can be accelerated almost for free. Directions it barely constrains, such as fine detail lost to blur or pixels hidden by a mask, are where momentum turns injected noise into large, slowly decaying excursions. Those excursions are the hallucinated content that the perception-distortion literature worries about (Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff).
For one mode with effective curvature $\lambda$, step size $\alpha$, momentum $\beta$, and injected per-step noise of variance $\sigma^2$, the heavy-ball recursion is a second-order autoregression. Its stationary variance is
$$V(\lambda,\beta) = \frac{(1+\beta)\,\alpha^2\sigma^2}{(1-\beta)\,\alpha\lambda\,\bigl(2(1+\beta)-\alpha\lambda\bigr)},$$
and its contraction factor is the spectral radius $\rho(\lambda,\beta)$ of the recursion. We take the effective curvature to be $\lambda_i = s_i^2/\sigma_y^2 + 1/v_i$. Here $s_i$ is the $i$-th singular value of the forward operator, $\sigma_y^2$ is the measurement noise variance, and $v_i$ is a Gaussian-prior variance for that mode, which approximates the denoiser's shrinkage and is fit from a natural-image power spectrum. Given a budget $\tau$, which is a multiple of the variance plain gradient descent would have in that mode, we set
$$\beta_i(\tau) = \arg\min_{\beta\in[0,1)} \rho(\lambda_i,\beta) \quad \text{subject to} \quad V(\lambda_i,\beta) \le \tau\, V(\lambda_i,0).$$
This is a one-dimensional problem per mode and is solved once per operator. Setting $\tau=1$ recovers unaccelerated behaviour. Raising $\tau$ spends the extra variance first where acceleration buys the most convergence per unit of amplification.
Relation to Prior Work
Three recent works motivate the proposal by showing that iteration dynamics in diffusion solvers set the perception-distortion operating point. Improving Diffusion-based Inverse Algorithms under Few-Step Constraint via Linear Extrapolation (LLE) learns linear combinations of past iterates, with coefficients split between the range space and the null space, from 50 training images. It treats extrapolation as a fidelity rescue rather than a frontier dial. We differ in two ways: our coefficients come in closed form from a noise model, and they resolve the full singular spectrum instead of two subspaces. NFCDS shows that low-frequency injected noise breaks fidelity while high-frequency noise produces texture, and it filters the noise with a static Fourier mask. It uses no momentum, and it works in the Fourier basis rather than the operator's own basis. Stage-wise Distortion–Perception Traversal in Zero-shot Inverse Problems with Diffusion Models (MAP-RPS) moves along the trade-off by re-noising an unaccelerated MAP estimate to a timestep $t_0$. Our knob $\tau$ plays the role of $t_0$, but it acts inside the iteration rather than as a two-stage restart.
Two works supply the stabilization idea. Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA shows that damping FISTA's over-relaxation prevents error blow-up under inexact steps. Universal intermediate gradient method for convex problems with inexact oracle interpolates between stable and fast methods with a power parameter. Both use a single global damping and measure success by objective value. We make the damping depend on the mode and measure success with perceptual metrics.
One theoretical result constrains the design. The FISTA stability analysis obtains its guarantees through ergodic averaging, meaning averaging the iterates over time. Averaging plausible images produces the blurry conditional mean that EnhanceNet identifies as fatal to realism. We therefore deliberately do not average. We bound variance by choosing the coefficients and accept last-iterate dynamics. Accelerated gradient methods with biased gradient estimates adds a further caution: momentum magnifies systematic bias exponentially. A biased denoiser could therefore defeat a variance-only budget, and we test for this directly.
Research Program
The method acts on the outer loop of zero-shot solvers that already work in the operator's singular basis, such as DDNM-style range/null decompositions. It is applied to deblurring with circular blur (whose singular basis is the Fourier basis), 4× super-resolution, and box and random inpainting, where the null-space modes have $s_i=0$ and so only the prior term constrains them.
The work has three parts. First, we fit $v_i$ per dataset and calibrate $\sigma^2$ from the solver's own noise schedule, so the method runs without training. Second, we check the model: across 64 seeds per image, we measure the empirical variance of each mode's iterate and compare it with $V(\lambda_i,\beta_i)$. If the linear-Gaussian model misses by more than a small constant factor in the ill-conditioned modes, the rest of the argument does not stand. Third, we sweep $\tau$ at 3, 5, and 10 network evaluations and plot PSNR against LPIPS and FID. To separate useful texture from hallucination, faces are additionally scored with an identity-embedding similarity to the ground truth, since identity drift is the recognized failure of strong generative priors (VQFR).
Baselines
The naive reference is the unaccelerated solver at the same step count, which is plain DDNM. LLE reports 16.64 dB PSNR for 3-step DDNM on CelebA-HQ inpainting.
The strongest competitor in the few-step regime is LLE itself, which reports 24.38 dB for the same setting. It is the most direct rival because it also extrapolates over past iterates and also separates constrained from unconstrained directions. We will run LLE's released procedure with its 50-image training set, on the same backbone, operators, and evaluation images. We also give it a tuning budget equal to our calibration time. Beating LLE means matching its PSNR at equal steps with no training, and dominating its point on the PSNR–LPIPS plane somewhere along our $\tau$ sweep.
For frontier traversal, the baseline is MAP-RPS with a $t_0$ sweep at matched network evaluations. The file reports no scalar PSNR or FID for MAP-RPS, only latency (23 s versus more than 250 s for Latent-DPS on MS-COCO). We will therefore compare frontier area rather than a quoted number.
For ImageNet 4× super-resolution, we include CDDB as an external reference point: 26.41 dB PSNR and 19.88 FID at 20 steps. It is a trained bridge model, so this is a ceiling rather than a like-for-like comparison.
Two ablations isolate the key ingredient. The first uses one global $\beta$, grid-searched per $\tau$. The second uses our per-mode budget but with the modes defined in the Fourier basis rather than the singular basis.
Why not just…?
Why not just learn the extrapolation weights, as LLE does? This is the real threat. LLE might simply learn the same allocation from data. We will regress LLE's learned range and null coefficients against our analytical $\beta_i$, averaged over each subspace. If they agree, our contribution reduces to an explanation of LLE plus training-free transfer to unseen operators and noise levels. We will test that transfer by changing the blur width and the mask ratio without refitting either method, which is where LLE's static weights are expected to degrade.
Why not just filter the injected noise in Fourier space, as NFCDS does, and use ordinary momentum? For circular deblurring the Fourier and singular bases coincide. There, NFCDS combined with a single global momentum may already capture most of the gain. NFCDS reports roughly 0.44 dB PSNR and 0.02 LPIPS improvement on CelebA-HQ denoising. We expect the two approaches to separate on inpainting and super-resolution, where conditioning is not a function of spatial frequency. The Fourier-basis ablation tests exactly this. If it matches the singular-basis version on inpainting, the operator-specific part of our method is unnecessary and we will say so.
Why not just tune one global momentum together with a re-noising timestep? A global $\beta$ must be set by the worst-conditioned modes. It therefore under-accelerates the well-measured directions or over-amplifies the poorly measured ones, as the spectrum dependence in the robustness analysis predicts. The global-$\beta$ ablation, swept jointly with $t_0$ at equal step counts, measures how much frontier area this loses.
Decisive Checks
The idea survives only if three things hold. First, predicted and measured per-mode variances agree in the ill-conditioned modes. Second, at 3 to 5 steps, the $\tau$ sweep matches or dominates LLE's CelebA-HQ inpainting point without training and beats global momentum on FFHQ and ImageNet super-resolution and inpainting. Third, identity similarity on faces degrades more slowly along our sweep than along the $t_0$ sweep at equal LPIPS. If the variance model holds but identity drifts faster, that would suggest the denoiser's bias rather than its variance is driving hallucination. The budget would then need a bias term, as the risk-sensitivity analysis anticipates.
References
- Robustness of Accelerated First-Order Algorithms for Strongly Convex Optimization Problems
- Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
- Improving Diffusion-based Inverse Algorithms under Few-Step Constraint via Linear Extrapolation
- NFCDS: A Plug-and-Play Noise Frequency-Controlled Diffusion Sampling Strategy for Image Restoration
- Stage-wise Distortion–Perception Traversal in Zero-shot Inverse Problems with Diffusion Models
- Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA
- Universal intermediate gradient method for convex problems with inexact oracle
- EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
- Accelerated gradient methods with biased gradient estimates: Risk sensitivity, high-probability guarantees, and large deviation bounds
- VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
- Direct Diffusion Bridge using Data Consistency for Inverse Problems