Letting the Measurements Decide: Residual-Tested Routing for Latent Unfolded Restoration

Modern image restoration increasingly runs inside the compressed "latent space" of an autoencoder, a small feature grid that stands in for the full-resolution image. This is attractive because it cuts compute by an order of magnitude or more. These systems still hide a stack of hand-set knobs: how many refinement steps to run, how strongly to trust the observed data versus a learned image prior, and how much generative texture to add. Those knobs are usually tuned once per dataset and then applied uniformly to every region of every image. That is wasteful and often wrong, because a single photograph can contain sharp, well-measured edges next to masked holes or heavily blurred patches.

We propose to replace these knobs with a local statistical test. At every refinement step, and for every latent token, the method checks whether the leftover mismatch between the observed image and the current reconstruction looks like pure sensor noise. It also checks how strongly the measurements constrain that token at all. The two answers route the token. Tokens whose residual is already noise-like and well constrained stop computing. Tokens whose residual still has structure go to a data-fidelity expert. Tokens the measurements cannot constrain go to a generative expert. The routing is conditional computation, but the gate is a test on the physics of the measurements rather than a learned router. The goal is a latent restoration network with no global step count and no perception-fidelity slider, and whose compute and prior strength both track what the data actually says.

Core Claim

For inverse problems with a known forward operator $A$ (the physical process that maps a clean image to what the sensor records, such as blur, downsampling, masking, or compressive projection), a per-token residual test in the measurement domain is a routing signal for conditional computation inside a latent unfolding network. We claim it has three properties a learned router lacks. First, it stays calibrated when noise level or degradation layout shifts away from training, because the test depends on the noise model rather than on the training distribution. Second, it cannot collapse onto a few experts, because nothing about it is learned. Third, it removes the stage count and the fidelity-versus-perception balance as tuning parameters.

We do not claim it beats a learned router on in-distribution data. We expect rough parity there, and the proposal is built to test where the two diverge.

What Exists and What Is Missing

Three lines of work each supply one piece. The first is deep unfolding, which turns an iterative optimizer into a fixed stack of network layers with learned step sizes. ADMM-Net does this for MRI and DGUNet for general restoration. ScaoedNet, UnfoldLDM and LADE-DUN move unfolding into latent spaces or add latent diffusion priors to counter the over-smoothing of regression-trained unfolding. All five fix the number of stages at training time, and all run every spatial location through every stage. We reuse their unfolded proximal-gradient structure and their evidence that a generative latent prior is needed for texture. We differ by making both depth and prior strength vary per token.

The second line is tuning-free inverse solvers, which motivate our routing signal. Wang et al. check whether the ADMM residual is white Gaussian noise to set the penalty parameter and to decide when to stop, but they apply this globally, to a whole hyperspectral cube. Peng et al. and LFlow replace heuristic diffusion guidance scales with analytically derived covariances. LFlow does this directly in a VAE latent space, but it spends roughly 267 seconds per ImageNet image with uniform computation over all latents. We take the whiteness idea from Wang et al., localize it to latent tokens, and use it to route rather than only to stop. We borrow LFlow's first-order linearization of the decoder to carry measurement gradients back into the latent.

The third line is conditional computation for restoration, which supplies the expert design and a constraint. AdaptIR shows that identical experts behave like the same frequency filter whatever the degradation. It reports that heterogeneous experts beat the homogeneous adapter FacT by 1.78 dB on Urban100 with low resolution plus noise. This is why our two experts are deliberately different in kind. AdaptIR's gate is learned and runs in pixel-space transformer features. Latent Harmony is the clearest example of the tuning burden we target: its fidelity-versus-perception balance is a single global slider $\alpha$ that a person drags at test time. Our routing turns that global slider into a per-token decision made by a statistic.

Research Program

The model starts from a pretrained VAE with encoder $E$ and decoder $D$, working on a latent grid $z$ downsampled from the image. Each unfolded stage $k$ first takes a data-consistency step. The measurement gradient is pulled into latent space through the decoder Jacobian $J_D$, as in LFlow:

$$\tilde z_k = z_k - \eta_k\, J_D(z_k)^\top A^\top\big(A D(z_k) - y\big),$$

where $y$ is the observation and $\eta_k$ is a learned step size. The model then applies a per-token proximal operator chosen from two experts. The fidelity expert is a small convolutional regression block in the style of DGUNet's proximal module, cheap and conservative. The generative expert is a few-step latent prior, obtained by consistency-distilling a latent diffusion model in the manner of Decorruptor, conditioned on $\tilde z_k$. A third option is exit: the token is frozen and skips all further computation.

Routing uses two local quantities, computed for each token $i$ over its footprint $\Omega_i$ in measurement space. The first is a residual test. Take the residual $r = y - A D(z_k)$ and the noise level $\sigma$, which is either known or estimated from residual high frequencies with a median-absolute-deviation estimator. From these we compute a normalized energy $e_i = \|r_{\Omega_i}\|^2 / (|\Omega_i|\sigma^2)$ and a whiteness score. The whiteness score is the largest normalized autocorrelation of $r_{\Omega_i}$ at small nonzero spatial lags:

$$w_i = \max_{\delta \neq 0,\,|\delta|\le 2} \frac{\big|\sum_{p} r_p\, r_{p+\delta}\big|}{\sum_p r_p^2}.$$

The second quantity is an observability score $o_i$. It is the gain of $A D$ along the token's latent directions, estimated with a couple of random Jacobian-vector products. It measures how much the measurements can see this token at all: it is near zero under an inpainting mask and small under strong blur.

The rule is fixed. A token exits when $e_i$ is near one and $w_i$ is below a threshold set by the null distribution of white noise over $|\Omega_i|$ pixels. It goes to the generative expert when its residual passes that test but $o_i$ is low, meaning the data are silent and only the prior can fill in detail. Otherwise it goes to the fidelity expert. A maximum depth caps runtime. The thresholds come from a chosen false-alarm rate on synthetic white noise, not from validation images, so no image-quality metric is ever tuned. Tokens that exit early can still be revisited if a neighbor's update changes their residual, since the test is cheap and rerun every stage.

Training has to make the network valid under any exit pattern. We therefore train with randomized per-token exit and expert assignments, mixed with the rule-driven assignments, using an L1 loss plus a perceptual loss on generatively routed regions. The degradations are spatially nonuniform: spatially varying blur kernels, spatially varying noise levels, and random partial masks. Otherwise per-token routing has nothing to exploit. Because routing is gather-and-scatter over a small latent grid, we will implement it with packed active-token batches and report wall-clock time on a single GPU alongside FLOPs. The systems literature on MoE (systems-aware MoE) warns that theoretical FLOP savings often fail to become real speed.

The analysis program asks three things. Does the routing map recover the true structure of the degradation? That is, does the generative-expert region match the mask or the strong-blur zones, and does exit happen first in flat, well-measured areas? Does compute per image scale with degradation severity? And where does the method break? Spatially correlated noise, as after a camera's processing pipeline, is the obvious failure mode, because a whiteness test assumes white noise.

Baselines

The naive reference is the same latent unfolding network run densely with a fixed stage count $K$ and a fixed global blend between the two experts, analogous to Latent Harmony's $\alpha$. $K$ and the blend are chosen by grid search on a validation split. The grid search gets the same number of validation evaluations our method spends calibrating thresholds on noise, and the trained weights are identical. Beating it means equal or better PSNR and LPIPS at lower average compute, or better quality at matched compute. The second condition matters most on test degradations outside the validation range, where a single global setting has no reason to transfer.

The strongest competitor on the central question is the same architecture with a learned router: a top-1 gate over {fidelity, generative, exit} trained with a load-balancing loss and a compute penalty in the style of adaptive halting. It uses the same data, the same training budget, and a comparable hyperparameter sweep over the penalty weight. This is the direct test of whether a physics-based gate adds anything. We will also run a variant that feeds $e_i$, $w_i$ and $o_i$ to the learned gate as extra inputs, which separates "the statistics are useful features" from "the statistics should be the decision." Beating the learned router means a better quality-compute trade-off under held-out noise levels and degradation layouts. Parity in distribution is the expected outcome and is acceptable.

Among existing published methods, the closest latent training-free solver is LFlow, run with its released settings on FFHQ and ImageNet at 256×256. The file reports only relative figures for it: an 8.8% LPIPS improvement over the next best method on FFHQ Gaussian deblurring, and an average runtime of 267 seconds per ImageNet image against 705 for PSLD. For FFHQ inpainting, the strongest absolute number in the file is LPIPS 0.0794 from Peng et al., against 0.1323 for DPS. Beating these means matching or improving LPIPS at one to two orders of magnitude less runtime. That is plausible because our method is a trained feedforward unfolding, not a per-image ODE solve, so the comparison also has to state that ours requires training while theirs does not.

For compressive sensing on Set11 and BSD68, we compare against pixel-space DGUNet. The file reports it as up to 2 dB better than OPINE-Net+ at a 1% sampling ratio, with no absolute PSNR given. We will retrain it on our splits. For hyperspectral deconvolution on CAVE, we compare against the global whiteness-tuned ADMM of Wang et al., which the file reports as 3–5 dB better than HLP and 3DFTV. No reported result was found on Lacuna for UnfoldLDM on any of our benchmarks, so we re-implement a dense latent-unfolding-with-diffusion-prior variant as the representative of that family.

Two ablations isolate the key ingredient. The first is global residual-test routing, where the whole image exits or switches expert together as in Wang et al.'s per-image stopping; it isolates the value of acting per token. The second drops the observability score, so routing uses only $e_i$ and $w_i$; it isolates whether choosing between prior and data needs a separate "can the data see this" signal.

Why not just…?

Why not just grid-search the stage count and the fidelity-perception balance once per task? Uniform settings are known to be brittle across noise levels and operators. Peng et al. show heuristic guidance in DPS and ΠGDM leaves FFHQ inpainting at LPIPS 0.1323 and 0.1422, against 0.0794 once the covariance is derived. Wang et al. report 3–5 dB losses for fixed-penalty baselines on CAVE. But these results concern global mis-tuning, not spatial non-uniformity. Our naive baseline tests the latter directly: if a well-tuned global $K$ and blend match per-token routing on spatially varying degradations, the per-token claim fails.

Why not just use a learned router, as every MoE does? This is the real threat. On in-distribution data a learned gate may match or beat a fixed test, and we say so. The argument for the residual test rests on shift and stability. Learned routers are prone to collapse and need balancing losses, as the MoE literature summarized in GShard and V-MoE discusses. And a gate trained on one noise range has no mechanism to know that a heavier-noise token is already "done." The held-out-noise and held-out-layout experiments settle this. If the learned router with the statistics as inputs wins even under shift, we will report that the statistics are best used as features, not as the gate.

Why not just adapt per image, as Wang et al. do, instead of per token? Per-image adaptation already removes much of the tuning. If most of the gain comes from choosing one depth and blend per image, token routing and its systems cost would not be worth it. The global-routing ablation measures exactly this. We expect the gap to be near zero on spatially uniform degradations and to open on mixed masks and spatially varying blur, and we will report both regimes.

Why not just port AdaptIR's heterogeneous experts into the latent space? That would supply the heterogeneous experts we agree are needed. It keeps a learned gate and a fixed depth, however, so it addresses neither tuning of the stage count nor the prior-versus-data balance. In our design it corresponds to the learned-router baseline with AdaptIR-style experts. We include it as a variant to check that our gains are not simply better experts.

Proposed Experiments

The main evaluation spans FFHQ and ImageNet at 256×256 (Gaussian and spatially varying deblurring, 4× super-resolution, and inpainting with random and box masks), Set11 and BSD68 compressive sensing at 1–25% sampling, and CAVE hyperspectral deconvolution. Each task has a spatially nonuniform variant built by blending two degradation levels across random regions. Metrics are PSNR, SSIM and LPIPS, together with average latent-token-stages per image, FLOPs, and measured wall-clock time. Every method is plotted as a quality-versus-compute curve rather than a single point.

The first decisive check is a shift test. We train on noise σ in one range and one family of mask shapes, then test on higher σ and unseen layouts. The prediction is that the residual-tested model degrades gracefully and spends more compute on harder inputs, while the learned router misallocates. The second check measures the overlap between generative-expert routing and ground-truth masks and blur-strength maps, which asks whether the routing does what the physics says it should. The third probes the known limit: we inject spatially correlated noise and report how far the whiteness test's false-alarm rate drifts and how much quality falls. A negative result there defines where the method should not be used.

Finally, a UHD stress run on UHD-Blur tests whether token-level savings survive at 4K, where DreamUHD is the reference. Because the blur there is blind, we use an estimated operator and treat this run as exploratory rather than as part of the core claim.