Compressing a trained image network to 4 or 2 bits without retraining usually works one block at a time. A block is a small group of layers, such as a residual unit or an attention-plus-MLP unit. The method freezes the original network, runs about a hundred unlabeled images through it, and adjusts the low-precision block until its outputs match the original's. The standard mismatch measure is plain mean squared error over every feature value. That measure treats an error in a smooth region of a feature map the same as an error on an edge, even though the rest of the network may amplify the two very differently. For classifiers, the established fix weights errors by how much they move the class prediction. Image restoration and generation have no class labels, so practitioners fall back to unweighted error. That is where super-resolution models lose texture at low bit-widths.
We propose a different weighting for image-to-image networks. Convolutional networks process every image position in roughly the same way, and their statistics, averaged over many images and positions, do not depend on location. Under those conditions, the matrix that says how a block's output error grows into final-image error decomposes along Fourier frequencies. It reduces to one small channel-by-channel matrix per spatial frequency. That matrix can be estimated cheaply from the frozen network with random probes and no labels. Block reconstruction then minimizes error in the Fourier domain, weighted by this measured downstream sensitivity. The frequency weighting is derived from how the network propagates errors, not hand-tuned. It is also far cheaper than the Fisher approximations now used for vision transformers.
Core Claim
Let $\Delta = f_{FP}(X) - f_Q(X)$ be the difference between a block's full-precision and quantized outputs on calibration images $X$. Let $g$ be the frozen remainder of the network that maps block output to the final image, and let $J$ be its Jacobian. To second order, the final-image squared error is $\Delta^\top \mathbb{E}[J^\top J]\,\Delta$. Spatial mean squared error replaces $\mathbb{E}[J^\top J]$ with the identity. Our claim is that for image-to-image networks this expected matrix is approximately translation-invariant. If so, it is diagonalized by the 2D discrete Fourier transform up to the channel dimension, and the second-order error becomes
$$ \mathcal{L}_{block} \;=\; \sum_{\omega} \hat{\Delta}(\omega)^{H}\, G(\omega)\, \hat{\Delta}(\omega), $$
Here $\omega$ ranges over spatial frequencies, $\hat{\Delta}(\omega)\in\mathbb{C}^{C}$ is the Fourier coefficient of the error across the block's $C$ channels, and $G(\omega)$ is a $C\times C$ Hermitian sensitivity matrix for that frequency. We call $G$ the spectral Gauss-Newton matrix.
This loss has several properties the alternatives lack. It is label-free, because $G$ is defined through the final-image metric (pixel error, which is what PSNR measures) rather than class log-probabilities. It handles amplitude and phase jointly, because it acts on complex coefficients. It captures cross-channel coupling at each frequency, which a diagonal Fisher approximation drops. The approximation itself can be tested directly by measuring how much energy of the probed $\mathbb{E}[J^\top J]$ falls off the frequency diagonal.
Research Program
This proposal sits between two literatures that have not been joined, and it borrows its mechanism from one to fix a stated weakness of the other.
The first literature, block reconstruction, supplies the method we reuse. BRECQ uses a second-order Taylor argument to show that cross-layer curvature concentrates inside structural blocks. It then weights block-output errors by a diagonal Fisher matrix computed from classification gradients. FIMA-Q shows that the diagonal discards cross-token and cross-channel structure. It adds a low-rank correction, at a reported cost of roughly 8 hours to calibrate Swin-B versus roughly 3 hours for simpler methods. Both methods define importance through class labels and in spatial or token coordinates. We keep their second-order logic but change the basis to Fourier and the target to output-image error. That substitution makes the structure cheap rather than expensive.
The second literature motivates the gap. Focal Frequency Loss shows that spatial losses under-penalize high-frequency error. It reweights Fourier coefficients by $|F-\hat F|^\alpha$, but only as an end-to-end training loss on final images. Wavelet methods such as HQG-Net and the sub-band normalization of Zhong et al. rebalance bands with fixed rules. None of these is used inside a block's calibration objective. All of them choose the weighting by heuristic, whereas we measure it.
Super-resolution post-training quantization is the application regime. Two lines of work constrain the design. PaC with density-based clipping quantizes EDSR and SRResNet with a pixel-by-pixel feature-transfer loss on clipping bounds. 2DQuant quantizes SwinIR down to 2 bits with spatial feature distillation. 2DQuant also reports that 2-bit models can suppress aliasing artifacts present in the full-precision model. That finding is a warning that matching the full-precision spectrum is not automatically matching the ground truth. We therefore report fidelity to both.
The work has four parts. The first is an estimator for $G(\omega)$. For each block, we inject random Gaussian probes $\delta$ into the block output on calibration images. We compute $J^\top J\delta$ with one forward-mode and one reverse-mode pass through the frozen remainder, then accumulate $\widehat{J^\top J\delta}(\omega)\,\hat\delta(\omega)^H$. Because white probes have flat spectra, this converges to $G(\omega)$ under the translation-invariance assumption. We bin frequencies radially and by orientation. We store $G$ per bin as diagonal plus rank-$r$, so cost scales with the number of bins rather than the feature-map size.
The second part is a diagnostic. On real models, we measure the fraction of probed curvature energy that lies on the frequency diagonal. The models are EDSR and SRResNet, which are convolutional with no batch normalization, and SwinIR, whose shifted-window attention breaks exact equivariance. This tells us where the method should and should not work before any accuracy numbers are run.
The third part plugs $\mathcal{L}_{block}$ into a standard block-reconstruction loop. Weights are learned with adaptive rounding, and clipping bounds are initialized by the density-based scheme of the PaC paper, so that only the loss changes.
The fourth part is a second variant whose target metric is a frequency-weighted output loss rather than pixel error. It separates two effects: the gain from propagating the right weighting back to blocks, and the gain from changing what "good output" means.
Baselines
The naive reference point is unweighted block reconstruction with spatial mean squared error. It uses the BRECQ-style block granularity, adaptive rounding, and density-based clipping initialization, but sets $G(\omega)$ to the identity. Every other comparison is measured against it, and it isolates whether any weighting helps.
The strongest existing convolutional baseline is DBDC plus PaC from Toward Accurate Post-Training Quantization for Image Super Resolution. That paper reports a 2.091 dB PSNR increase on Urban100 for EDSR×4 at 4 bits with 100 unlabeled images, and 31.146 dB PSNR on Set5 in the same setting. The strongest transformer baseline is 2DQuant, which reports a 4.52 dB PSNR increase over prior methods on Set5 for SwinIR×2 at 2 bits. The file gives no absolute 2DQuant PSNR, so no reported absolute result was found on Lacuna for SwinIR at 2 bits, and we will reproduce it. We run both baselines with their published recipes on the same calibration images: 100 DIV2K images for EDSR and SRResNet, and the DF2K subset for SwinIR. We also run them with our block-wise weight rounding enabled, so that a gain cannot be credited to optimizing weights instead of only clipping bounds.
The curvature baseline is a label-free diagonal Fisher variant of BRECQ. It computes the diagonal of $\mathbb{E}[J^\top J]$ in spatial coordinates using the same probes and the same output pixel-error target. It answers whether frequency structure matters, or whether any measured curvature suffices.
The obvious-idea baseline adds Focal Frequency Loss on block outputs to the spatial loss. We tune $\alpha$ and the mixing weight on a held-out calibration split with the same tuning budget we give our own rank and binning choices. The output-only baseline applies Focal Frequency Loss to the final image during calibration while keeping spatial block losses.
The key ablations remove one ingredient at a time. One keeps only a scalar per frequency, dropping cross-channel terms. One uses $G$ estimated on a different image set. One uses a fixed high-pass Gaussian-residual filter in the style of Frequency-Supervised MR-to-CT Image Synthesis.
Calibration iterations, batch size, and wall-clock limits are matched across all methods, and probe-estimation time is charged to our method. Beating a baseline means a PSNR gain on Urban100 and Manga109 that holds across three calibration seeds and exceeds seed variance, together with lower output spectral error in the high-frequency bins. A gain of at least 0.2 dB over reproduced PaC at 4 bits, and a larger gap at 2–3 bits, is the bar we set in advance.
Why not just…?
Why not just add Focal Frequency Loss to each block's output? This is the obvious combination and the main threat. Its weighting $|F-\hat F|^\alpha$ responds to where the current error is large, not to where the downstream network amplifies error. Large feature-map error at a frequency the upsampler suppresses is wasted effort. It also has no cross-channel term. If the tuned heuristic matches our method within seed noise, the honest conclusion is that the contribution shrinks to "a tuning-free, derived weighting." The ablation that gives the heuristic an equal tuning budget is designed to show exactly this, and we will report it either way.
Why not just put a frequency loss on the final output and keep spatial block losses? Block reconstruction exists because end-to-end objectives overfit tiny calibration sets, which is the argument of BRECQ. An output loss also only reaches early blocks through the whole downstream network. Our method moves the output metric's preferences into each block through $G(\omega)$ while keeping local optimization. The output-only baseline tests whether this matters. It is a plausible competitor at 4 bits and less plausible at 2 bits, where per-block error is large.
Why not just use a diagonal Fisher matrix computed from output error, as BRECQ does for classifiers? A spatial diagonal assumes errors at different positions and channels contribute independently. Convolutional downstream networks mix neighboring positions, so the true matrix is dense in space but nearly diagonal in frequency. A spatial diagonal cannot represent "this block's errors at edge frequencies are amplified." The label-free diagonal Fisher baseline directly measures the cost of choosing the wrong basis. Its compute is matched to ours, so cost cannot explain the difference.
Why not just do quantization-aware training with a spectral loss? The PaC paper reports that quantization-aware training of EDSR takes about 8 days on a high-end GPU, versus minutes for post-training calibration on 100 unlabeled images. Our setting is the one where the full training data and that budget are unavailable. We do not claim to beat quantization-aware training, only to narrow the gap to it at a post-training cost.
Proposed Experiments and Decisive Checks
The models are EDSR×4, SRResNet×4, and SwinIR×2/×4. They are quantized at W4A4, W3A3, and W2A2 and evaluated on Set5, Set14, B100, Urban100, and Manga109. We report Y-channel PSNR and SSIM, calibration wall-clock time, and two radially binned log-power-spectrum errors: against the full-precision model and against the high-resolution ground truth. The two-reference report addresses the aliasing observation in 2DQuant.
Three checks decide the project.
The first runs before any accuracy experiment. If less than roughly 70% of probed curvature energy lies on the frequency diagonal for EDSR, the core approximation fails and we stop or switch to a local-window Fourier basis. We expect SwinIR to score lower. If our gains shrink there in proportion to that score, the explanation is supported.
The second check concerns predictive validity. We sample many candidate quantizations of a single block, with different roundings and clipping bounds, and record the actual final-image PSNR change for each. We then compute rank correlations with spatial error, diagonal Fisher error, Focal-Frequency block loss, and $\mathcal{L}_{block}$. The method is justified only if it ranks candidates better.
The third check is the full comparison against reproduced PaC and 2DQuant at matched budgets, with the gains concentrated in the texture-heavy Urban100 and Manga109 sets and in high-frequency spectral bins.
As a stretch test of generality, we apply the same estimator to diffusion U-Nets under the timestep-sampling protocol of PTQ4DM. That paper reports an Inception Score of 9.55 at 8 bits on CIFAR-10, against 9.28 for full precision. Here $G(\omega)$ is estimated separately per timestep range, which tests whether the measured sensitivity shifts toward high frequencies as $t\to 0$, the way the coarse-to-fine picture of the reverse process predicts.
References
- BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
- Focal Frequency Loss for Image Reconstruction and Synthesis
- HQG-Net: Unpaired Medical Image Enhancement With High-Quality Guidance
- Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution
- Toward Accurate Post-Training Quantization for Image Super Resolution
- 2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution
- Frequency-Supervised MR-to-CT Image Synthesis
- Post-training Quantization on Diffusion Models