Operator-Defined Frequency Bands for Deblur-then-Upscale Restoration

When a blurry, low-resolution photo is fixed in two steps, first removing the blur and then enlarging it (super-resolution), the second step usually makes things worse in a specific way. Undoing blur means dividing by the blur's frequency response. Where that response is close to zero, the division boosts noise and produces ringing. The upscaling network was trained on clean images, so it reads that ringing as real fine detail and sharpens it. This proposal starts from a simple observation: the frequencies where the damage happens are known before any network runs. They can be computed in closed form with a Fast Fourier Transform (FFT) from the blur kernel, the downsampling factor, and the noise level. We propose passing that map between stages. Each stage then knows, frequency by frequency, which content was actually measured and which content has to be invented. Measured frequencies are held to the data. Unmeasured frequencies are handed to a generative model, which is free to make them look realistic.

This matters for more than error propagation. It also changes how quality is scored. Current perceptual metrics reward sharp detail whether or not the detail agrees with the input photograph. Blau et al.'s perception-distortion tradeoff says realism and low error cannot both be maximized. It does not say every hallucination is equally acceptable. Inventing texture in frequencies the camera never recorded is legitimate. Contradicting frequencies it did record is an error that metrics fail to catch. The same reliability map lets us score these two cases separately.

Core Claim

The claim is that for compound degradations (blur followed by downsampling, plus noise), the natural place to split an image into "keep faithful" and "free to synthesize" is the forward operator's own per-frequency signal-to-noise ratio. A fixed wavelet split into low and high bands is the wrong boundary. Let $A = SK$ be the composed operator, with $K$ the blur, $S$ the downsampling by factor $s$, $\sigma^2$ the noise variance, and $P(\omega)$ a natural-image power spectrum prior (roughly $1/|\omega|^2$). Define the reliability of frequency $\omega$ as the Wiener gain of the composed operator:

$$r(\omega) = \frac{|\hat A(\omega)|^2 P(\omega)}{|\hat A(\omega)|^2 P(\omega) + \sigma^2},$$

Here $\hat A(\omega)$ is the effective frequency response after accounting for aliasing. Frequencies above the low-resolution Nyquist limit and near-zeros of the blur kernel get $r \approx 0$. Frequencies that survived both degradations get $r \approx 1$. A wavelet high band mixes both kinds. For motion blur, whose spectrum has zero lines at every scale, some "high" frequencies are well measured and some "low" ones are destroyed. Methods such as WGSR, which restricts the adversarial discriminator to wavelet high subbands for bicubic SR, and WaveFace, which runs diffusion only on the low wavelet band for faces, supply the method template of "different losses for different bands." Both use a band boundary that ignores the operator. That is harmless for bicubic downsampling, whose response falls off smoothly. We expect it to be harmful for blur composed with downsampling.

Research Program

The system has three parts.

The first is a closed-form spectral analysis step. It computes $r(\omega)$ and a regularized inverse estimate $x_0$ for the composed operator using FFTs. This is the classical frequency step of half-quadratic splitting, which alternates a per-pixel prior step with a global linear solve that the FFT diagonalizes. To keep the periodic-boundary assumption from producing wrap-around ringing, images are edge-tapered and padded before the solve. We report the residual boundary artifact separately.

The second part is a feed-forward generator $g$ conditioned on $x_0$ and on $r$ (broadcast as a spectral feature). It is trained with a split loss. On the reliable band it minimizes an $\ell_1$ distance to ground truth after filtering both images by $r$. On the unreliable band it is judged only by an adversarial discriminator and a perceptual loss applied to content filtered by $1-r$. This is WGSR's idea with the operator-defined boundary substituted for the wavelet one.

The third part is a final data-consistency projection, again solved in closed form with FFTs:

$$\hat x = \arg\min_x \|Ax - y\|^2 + \mu \|x - g(x_0, r)\|^2.$$

Here $\mu > 0$ is a single scalar dial. Small $\mu$ pins the measured band to the data. Large $\mu$ trusts the generator. Sweeping $\mu$ at test time traces a perception-distortion curve without retraining. This addresses the controllability gap noted for fixed-operating-point models, and it does so more cheaply than time-based early stopping in DiracDiffusion, which needs a model trained per operator.

The analysis half of the program uses $r$ as an evaluation tool. We define measured-band PSNR as PSNR between $r$-filtered output and $r$-filtered ground truth. We define invented-band realism with no-reference and distributional metrics computed on $(1-r)$-filtered content. The metric-misalignment hypothesis can then be tested directly. If standard perceptual scores (LPIPS, NIQE, the Perceptual Index) improve for serial pipelines while measured-band PSNR drops, the metrics are rewarding contradictions of the recorded signal. This is the "hallucination blindness" the literature describes but has not isolated.

Relation to Prior Work

Two groups of papers motivate the problem. Gated Fusion Network showed that serial deblurring then SR amplifies ringing. It fixed this with an end-to-end spatial network trained on pixel losses, so it neither exploits a known kernel nor addresses realism. Joint Demosaicing and SR and RL-Restore found the same compounding with other operation chains. They answered it by joint training, which in RL-Restore's case uses a PSNR reward that pushes toward blur. None of them tells the downstream stage which content is trustworthy, and that is what we add.

A second group supplies the consistency machinery. DDNM replaces the range-space part of each diffusion estimate with $A^\dagger y$ and generates only the null space. It is our closest relative. Its split is hard, meaning range versus null space, and it has no explicit noise-aware per-frequency weighting. It also needs about 100 diffusion steps. Our split is soft and noise-aware, and the generator is single-pass. NPN learns null-space projections that speed up PnP and diffusion solvers, but it does not evaluate perceptual quality. DiffECC warm-starts diffusion from a CNN estimate and inherits that estimate's artifacts. That is exactly the propagation problem, left without a frequency-level remedy.

On the metric side, PAMSE weights error by a fixed human contrast-sensitivity low-pass filter. Re²IC builds a wavelet-Wasserstein distortion that is strict on low frequencies and lenient on high ones. Both use perception-defined bands. We argue that for restoration the leniency should follow what was measured, not what humans are sensitive to. The two bands can disagree, and measuring that disagreement is part of the study. LFc-SR balances perception and distortion with a DWT low-frequency constraint inside ADMM-based training, again for bicubic SR only.

Baselines

The naive reference is the serial pipeline. It runs FFT Wiener deconvolution with the true kernel, then an off-the-shelf generative upscaler (Real-ESRGAN), with no information passed between the two. A second naive point is the Wiener-plus-projection output with no generator ($g = x_0$), which marks the distortion-optimal end.

The strongest learned competitor for compound blur plus SR is GFN, retrained on our training data with the same degradation sampler. On LR-GOPRO it reports 27.74 dB PSNR (GFN). We will also train a GFN-sized network with an adversarial loss, so that the comparison is not only about perceptual training.

The strongest zero-shot competitor is DDNM+ run with the same composed operator $A$ and noise level. No reported result was found on Lacuna for DDNM+ PSNR or FID on compound blur plus SR. DiffECC reports 28.47 dB PSNR and 24.08 FID on FFHQ 4× SR (DiffECC). That is a single degradation, so we rerun it with our composed operator.

For the band-split idea itself, WGSR reports 26.33 dB PSNR and 3.428 NIQE on BSD100 4× bicubic SR (WGSR). We reproduce this number first, then retrain WGSR on the compound task.

All learned models share one generator backbone, the same DIV2K training images and degradation distribution, the same iteration budget, and the same hyperparameter-search budget. DDNM+ gets a sweep over its step count.

The key ablation swaps $r(\omega)$ for WGSR's fixed wavelet split and otherwise keeps our generator, losses, and projection. If this ablation matches the full method, the operator-defined boundary is not doing the work. A second ablation removes the $r$ conditioning but keeps the projection. A third removes the projection.

Beating a baseline means one of two things. Either our $\mu$-swept curve dominates the baseline's operating point, reaching lower LPIPS at equal or higher PSNR. Or it matches the baseline's realism with at least 1 dB higher measured-band PSNR. Either result must hold on at least two of the three test sets.

Why not just…?

Why not just train one large end-to-end network on the compound task? This is a real threat, and GFN already beats serial stacks this way. A spatial network trained with a perceptual loss has no mechanism for staying consistent with measured frequencies while freely inventing the rest. The measured-band PSNR diagnostic will show whether it quietly contradicts the data. If an adversarially trained end-to-end network matches our curve on both measured-band PSNR and realism, we will report that the closed-form structure buys only controllability and speed. Our $\mu$ sweep costs one FFT solve per setting, where an end-to-end network would need retraining for each tradeoff point.

Why not just fine-tune the upscaler on the deblurrer's outputs? This is RL-Restore's joint fine-tuning, and it may remove most ringing amplification. We include it as a baseline: Real-ESRGAN fine-tuned on Wiener outputs. It learns to suppress ringing on average. It cannot know which frequencies are unreliable for a given kernel, and that set changes sharply across motion-blur directions. We therefore test on kernels held out from training. A small gap on seen kernels together with a large gap on unseen kernels would confirm that the explicit map matters.

Why not just use DDNM+ with the composed operator? It already separates what was measured from what was invented. The differences we test are the soft, noise-aware weighting and inference speed. DDNM's hard range-space replacement copies noise-dominated components near the kernel's zeros straight into the output. Comparing at matched noise levels (σ from 0.01 to 0.05) should show DDNM+ degrading faster as noise grows. If it does not, our advantage reduces to runtime, and we will say so.

Proposed Experiments

The primary benchmark is synthetic compound degradation with known kernels. We apply random motion and defocus kernels, 4× downsampling, and Gaussian noise to DIV2K for training. Testing uses BSD100, Urban100 (dense grids, where motion-kernel zeros should cause visible moiré in serial pipelines) and Set14. The motion kernels are split so that the test kernels are unseen.

The stress test is LR-GOPRO, with per-image kernels estimated by a standard blind estimator. There the blur is spatially varying and the kernel is only approximate, so we expect our gains to shrink. We report this honestly against GFN's 27.74 dB. PSR4K serves as a realistic compound set evaluated by no-reference metrics only.

We report PSNR, SSIM, LPIPS, NIQE, measured-band PSNR, and invented-band FID. To test metric misalignment, we run a two-alternative forced-choice human study on 200 image pairs from Urban100. For each pair we check whether LPIPS, NIQE, and human preference each favor the output that violates the measured band more. We also compare how well $r$-filtered LPIPS and plain LPIPS predict human choices. As a sanity check, measured-band PSNR should correlate with human judgments on TID2013 blur distortions at least as well as PAMSE, whose reported Spearman correlation on TID2008 is 0.9162 (PAMSE).

The decisive check is the fixed-wavelet ablation on unseen motion kernels. If the operator-defined split does not beat the wavelet split there by a clear margin in measured-band PSNR at matched realism, the core claim is wrong. The method would then reduce to WGSR plus a projection.