When engineers prune an image-restoration network so it runs faster, they usually check that a pixel-error score such as PSNR (peak signal-to-noise ratio, a log-scaled mean squared error) barely moves. Pruning here means zeroing most of the network's weights, often in the "2:4" pattern that NVIDIA's sparse tensor cores (dedicated matrix hardware) can execute at up to twice the speed. That check is known to be the wrong one. Pixel-error scores reward blurry, averaged outputs, and pruning removes the small weights that plausibly carry fine texture. We propose that hardware-friendly sparsity makes restoration outputs too predictable: waxy, over-smoothed images that keep their PSNR while losing perceived quality. We also propose a cheap way to catch and correct this.
The detector comes from free-energy quality assessment. This is a family of metrics that scores an image by how much of it a simple predictive model of natural images fails to predict. Noise and artifacts raise this unpredicted "residual," while over-smoothing lowers it below natural levels. So the change in residual entropy (the information content of what the predictor misses) between a pruned network's output and its dense parent's output is a signed, label-free signal. It says whether sparsity is erasing texture or adding artifacts, and it costs a small linear predictor rather than a VGG forward pass. We will first test whether this gap is real. We will then use the signal to choose which layers get which sparsity pattern, and to regularize the fine-tuning that follows pruning. Speedups will be measured in wall-clock time, not theoretical FLOPs.
Core Claim
The obvious combination of these three directions is to speed up free-energy metrics themselves with structured sparsity. The file's candidate Gap 1 frames it this way, since the sparse-coding metric of A reduced-reference quality assessment scheme for blurred images is slow. We reject that idea. A skeptic would simply replace iterative sparse coding with a tiny dense convolutional predictor. At that size dense GPU kernels are already fast, so sparsity overhead is not the bottleneck. Our proposal inverts the roles.
Sparsity is the application regime: restoration networks compressed for deployment. The objective-subjective quality gap is the failure mode we predict. Free energy is the measurement and control signal. The claim has two parts. First, at matched PSNR loss, 2:4 and moderate unstructured pruning of restoration networks lowers the output's residual entropy relative to the dense model, and human raters notice this. Second, a pruning procedure that constrains this signed shift gives better perceived quality at the same measured speedup than procedures tuned on PSNR alone.
The motivating evidence comes from two places. Perceptual Losses for Real-Time Style Transfer and Super-Resolution explains why pixel-wise objectives push networks toward the conditional mean of plausible outputs, which looks blurred. Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination reports that its output, which it argues looks better, had a worse RMSE (44.90 vs 44.24 for its baseline). Neither paper studies compressed networks.
The sparsity side supplies the method. Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch trains 2:4 and 4:8 networks with a sparse-refined straight-through estimator (a gradient trick that stabilizes which weights stay pruned). ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization wraps a 2:4 core in block-diagonal matrices to reduce pruning error. Both evaluate only classification, translation, or language modeling. That is exactly the regime in which a texture-erasing failure would not be seen.
The Signal
Let $y_d$ and $y_s$ be the outputs of the dense and pruned networks on the same input. Following A dual-model approach to blind quality assessment of noisy images, fit a linear autoregressive predictor to each output, meaning a model that predicts each pixel from its local neighbourhood. Take $R(\cdot)$ to be the prediction residual and $H(\cdot)$ its entropy. The pruning-induced free-energy shift is
$$\Delta_{FE} = H\big(R(y_s)\big) - H\big(R(y_d)\big).$$
Negative values mean the pruned output is more predictable than its dense parent, which is our predicted over-smoothing signature. Positive values mean it contains unpredicted structure such as ringing or blocking.
Comparing against the dense parent rather than a pristine image does two things. It removes most of the content dependence that plagues blind residual-entropy metrics, because texture-rich scenes have high entropy regardless of quality. It also makes the signal computable on unlabeled calibration images.
For training we use a differentiable surrogate. Under a Gaussian residual assumption, entropy is monotone in $\log \mathrm{Var}(R)$, so we penalize $|\log \mathrm{Var} R(y_s) - \log \mathrm{Var} R(y_d)|$ alongside the usual distillation error $\|y_s - y_d\|^2$.
This deliberately reuses the simplest generative model in the free-energy literature. Structured Computational Modeling of Human Visual System for No-reference Image Quality Assessment reduces free energy to one scalar inside a 33-feature regressor with a VGG-16 stage. That is too heavy to sit inside a pruning loop, and it was never used as a training signal. A known constraint cuts against us: the dual-model paper validated residual entropy only on additive Gaussian noise, with hand-tuned switching thresholds. So we must show the signal tracks smoothing, not just noise, before relying on it.
Research Program
The work proceeds in three stages.
The first stage is a measurement study. We take restoration networks for JPEG artifact removal, starting from the AR-CNN setting of Compression Artifacts Reduction by a Deep Convolutional Network. We add denoising and super-resolution tasks drawn from the LowLevelEval Dataset, including at least one larger modern backbone so that the tensor-core speedup is non-trivial. We prune each network in 2:4, 4:8, and unstructured patterns at 50–90% sparsity. For each network we report five things: PSNR, $\Delta_{FE}$, a pixel-level learned full-reference score (BiRQA), the analytic PAMSE score, and a blind no-reference score. The stage succeeds if we observe a consistent negative $\Delta_{FE}$ and a perceptual-metric drop that is disproportionate to the PSNR drop.
The second stage validates the signal against humans. Existing IQA databases contain no pruning artifacts. We therefore run a pairwise human study on dense-versus-pruned outputs, matched for PSNR loss, and test whether the sign and size of $\Delta_{FE}$ predict which image raters prefer. We also check the signal on existing smoothing-heavy data: the blur subsets of LIVE and TID2013, and PIPAL, whose restoration artifacts include over-smoothing.
The third stage is control. The signal is used twice:
- Pattern allocation. A budgeted search assigns each layer dense, 4:8, or 2:4. It minimizes measured latency subject to a bound on $|\Delta_{FE}|$ over a calibration set.
- Recovery fine-tuning. The surrogate regularizer above is added to the post-pruning fine-tuning objective.
Both mechanisms apply equally to SR-STE training and to ARMOR-style post-training factorization.
Baselines
PSNR-calibrated 2:4 pruning. The naive reference point is magnitude 2:4 pruning with recovery fine-tuned under distillation MSE and layer choices made by PSNR sensitivity. This is standard practice. The SR-STE variant, per Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch, loses only 0.3 points of ImageNet top-1 (77.0% vs 77.3% dense). The dense restoration network is the quality ceiling. On LIVE1 at JPEG quality 10, AR-CNN reports 28.98 dB PSNR in Compression Artifacts Reduction by a Deep Convolutional Network. We found no reported result on Lacuna for pruned restoration networks, so all sparse numbers are ours.
VGG perceptual-loss recovery. This is the strongest competitor on the quality question: recovery fine-tuning with the VGG feature loss of Perceptual Losses for Real-Time Style Transfer and Super-Resolution, which directly targets the texture loss we predict. It gets the same data, the same sparsity masks, the same number of fine-tuning steps, and an equal hyperparameter search over its loss weight.
ARMOR factorization. This is the strongest structured-sparsity method, applied to the restoration network's convolutions reshaped as matrices, following ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization. That paper reports a 1.57x layer speedup against a 2.0x theoretical limit. We test it both with and without our regularizer.
Evaluation metrics and the "beating" criterion. For scoring we use BiRQA, which reports SROCC (Spearman rank correlation with human scores) of 0.988 on LIVE and 0.979 on CSIQ in BiRQA: Bidirectional Robust Quality Assessment for Images. We also use PAMSE, reported at 0.9162 on TID2008 in Perceptual Fidelity Aware Mean Squared Error. BiRQA's white-box fragility means we will not optimize against it. Beating a baseline means the following at equal measured A100 latency: a statistically significant human-preference win in the pairwise study, better BiRQA and PAMSE scores, and a PSNR within 0.1 dB.
Ablations. Three ablations isolate the key ingredient:
- a symmetric version of the regularizer against a one-sided version that penalizes only negative $\Delta_{FE}$;
- replacing the dense parent with the ground truth as comparator;
- replacing the autoregressive residual with a simple high-pass energy match, which would show whether "free energy" matters or any frequency statistic would do.
Why not just…?
Why not just use the VGG perceptual loss during recovery? This is the real threat, and we treat it as one. If VGG-loss recovery matches our regularizer on human preference, the free-energy contribution shrinks to three things: the diagnostic finding, a cheaper calibration signal, and a signed direction that VGG distances do not give. Human Preference Score: Better Aligning Text-to-image Models with Human Preference argues that ImageNet feature spaces favor surface texture over structure. That could cut either way here. The head-to-head comparison at an equal tuning budget settles the question, and we will report it whichever way it falls.
Why not just use 4:8 instead of 2:4? SR-STE's 4:8 ResNet-50 slightly exceeded the dense baseline (77.4% vs 77.3%). So a looser pattern might remove the perceptual loss by itself, at some cost to hardware support. The first stage includes 4:8 for exactly this reason. If 4:8 shows no negative shift, the allocation search reduces to "use 4:8 where the hardware supports it," and we will say so.
Why not just measure with BiRQA or PAMSE and skip free energy? For diagnosis alone, these might suffice, and we will report their agreement with $\Delta_{FE}$. But BiRQA needs a pristine reference and is gradient-fragile. PAMSE, a low-pass-filtered MSE, is designed to ignore high frequencies and is therefore poorly suited to detect their loss. Neither score says whether the damage is smoothing or artifacts. The high-pass ablation tests whether the autoregressive residual adds anything beyond a frequency statistic.
Why not just prune unstructured and run it fast with SMaT? High Performance Unstructured SpMM Computation Using Tensor Cores beat dense cuBLAS at sparsity down to 78%, but on banded and SuiteSparse matrices, not pruned convolution weights. The file also notes that its row-reordering targets static matrices. We include unstructured pruning at 80–90% with measured SMaT latency. This shows whether freeing the pattern removes both the speed penalty and the perceptual penalty.
Decisive Checks
The idea stands or falls on three checks.
The first check is whether PSNR-matched pruned outputs show a reliably negative $\Delta_{FE}$ that raters penalize. If pruning damage is instead artifact-like, or imperceptible, the premise fails.
The second check is whether $\Delta_{FE}$ predicts pairwise human preference among pruned variants better than PSNR does, and at least as well as BiRQA does. This is tested first on the LIVE and TID2013 blur subsets and on PIPAL, then on our own pruning study.
The third check is whether regularized allocation and recovery beat both PSNR-calibrated and VGG-loss recovery in human preference at identical measured latency on sparse tensor-core hardware. Speedup must be reported as wall-clock time, following ARMOR's lesson that theoretical 2x becomes roughly 1.57x in practice.
A negative outcome on the third check with positives on the first two still yields a useful result: hardware sparsity has a measurable perceptual cost that standard pruning evaluation cannot see.
References
- A reduced-reference quality assessment scheme for blurred images
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution
- Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- A dual-model approach to blind quality assessment of noisy images
- Structured Computational Modeling of Human Visual System for No-reference Image Quality Assessment
- Compression Artifacts Reduction by a Deep Convolutional Network
- BiRQA: Bidirectional Robust Quality Assessment for Images
- Perceptual Fidelity Aware Mean Squared Error
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- High Performance Unstructured SpMM Computation Using Tensor Cores