Some systems run small neural networks around a standard video or image codec such as HEVC or JPEG. A network before the encoder reshapes the input, and a network after the decoder repairs or restores the output, so an ordinary hardware codec delivers higher quality or carries extra information. The codec itself cannot pass gradients, because its rounding and bit-packing steps are discrete. These wrappers are therefore trained through a differentiable codec proxy, a smooth imitation of the codec, and the proxy is swapped for the real codec at deployment. The wrappers are expensive. The strongest sandwich models have about 8 million parameters and must run on the same edge device as the encoder. So far the only way to shrink them has been to hand-pick a narrower network. This proposal asks how such wrappers should be pruned. Pruning means deleting weights, and the choice of what to delete ranges from single scattered weights, through small fixed patterns such as "2 of every 4", up to whole channels. The choice is made on two grounds: which deletions the real codec tolerates, and which deletions the target chip can actually turn into speed.
The central observation is that in this setting pruning decisions are scored through the proxy, and the proxy is known to be wrong. The authors of the sandwich papers report an unresolved gap between proxy and real HEVC performance. We conjecture that this gap is not uniform across the network. It should concentrate in the layers that write the high-frequency, dither-like "neural codes" into the codec's input, because those are exactly the signals whose survival under real quantization and in-loop filtering the proxy models least faithfully. If that is true, then gradient-based importance scores computed through the proxy systematically mis-rank those layers. It also follows that the right pruning granularity differs between the pre-processor, whose last layers shape the codes, and the post-processor, which behaves more like an ordinary restoration network. We propose to measure this directly. Our fix is to rescore candidate pruning decisions with forward passes through the real codec, and then to allocate a granularity to each layer under a latency model of the specific deployment chip.
Core Claim
Two claims can be tested. The first is that the ranking of prunable weight groups by estimated loss increase agrees poorly between the proxy and the real codec for code-shaping layers, and agrees well for the rest of the network. The second is that, at matched measured latency on a given device, a wrapper pruned with real-codec rescoring and per-layer granularity reaches better rate-distortion performance than two alternatives: a hand-narrowed dense wrapper, and a uniformly 2:4-sparse wrapper. The first claim is the new scientific content. The second is what makes it useful. If the first claim fails and proxy and real rankings agree everywhere, the real-codec rescoring is unnecessary and the work reduces to granularity allocation. We would report that outcome as a negative result rather than argue around it.
Background and Positioning
The application regime comes from sandwiched compression. Sandwiched Video Compression and Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers wrap HEVC or JPEG with neural pre- and post-processors. They train through a composite proxy made of block DCT for intra coding, UFlow motion for inter coding, and a small U-Net standing in for the loop filter. They report roughly 6.5 dB PSNR gains for low-resolution transport and about 30% bitrate savings at equal LPIPS. These papers motivate the proposal in two ways: they document the proxy gap, and they contrast an 8M-parameter model with a 57k-parameter "Slim" model obtained by manual uniform shrinking, with no pruning. Two machine-vision preprocessors supply a second testbed and a second kind of proxy. Preprocessing Enhanced Image Compression for Machine Vision trains a preprocessor through a learned BPG proxy. A Preprocessing Framework for Video Machine Vision under Compression uses a smooth virtual H.264 codec and explicitly names computational overhead on edge devices as an open problem. The two proxy families, learned and analytic, may fail in different places. That difference is a useful contrast for our first claim.
The method comes from the pruning literature. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch supplies SR-STE, a training rule that stops pruned weights from flickering back on, together with a mask-stability metric. We reuse both, because a noisy proxy gradient should make mask flickering worse. DSA allocates sparsity per layer under a global budget with continuous relaxations. S2HPruner closes the gap between soft and hard masks by distillation. Both assume the loss being differentiated is the true loss. Our setting breaks that assumption: there is a second discretization gap, the codec's, sitting underneath the mask's. EigenDamage shows that pruning in a decorrelated basis keeps dense, hardware-friendly shapes. It is a candidate coarse-granularity option, but its curvature estimates would also be computed through the proxy. The hardware constraint comes from two sources. MUSTAFAR finds that unstructured sparsity pays off only with custom kernels. The survey of Cheng et al. notes that fixed 2:4 patterns accelerate only on specific GPUs. Encoder-side wrappers often run on mobile CPUs or NPUs, so the granularity that is "fast" depends on the device and must be measured, not assumed.
Research Program
We begin with trained High Complexity sandwich wrappers for two tasks: low-resolution transport through HEVC, and LPIPS-optimized perceptual coding. We also train a Lu-style detection preprocessor through a learned BPG proxy. We partition each network into prunable groups at three granularities: whole output channels, 2:4 blocks, and individual weights, the last two taken in tile-sized chunks.
The measurement study compares two quantities for each group $g$. The first is the proxy-estimated loss increase from removing it, computed with the usual first-order Taylor score through the proxy. The second is the real loss increase
$$\Delta_{\text{real}}(g) = L_{\text{real}}(\theta_{\setminus g}) - L_{\text{real}}(\theta),$$
where $\theta$ are the wrapper weights, $\theta_{\setminus g}$ are those weights with group $g$ zeroed, and $L_{\text{real}}$ is the rate-distortion (or task) loss measured by actually encoding and decoding a small calibration set with the real codec at fixed QP. This quantity needs only forward passes, so it can be computed for every channel group and for a random sample of finer groups. We report the Spearman rank correlation between the two quantities, broken down by layer depth and by pre- versus post-processor. Our prediction is low correlation concentrated in the final pre-processor layers.
The pruning method uses the proxy for cheap screening and the real codec for final decisions. The proxy nominates candidate groups in each layer. Real-codec forward evaluation rescores the top candidates, and a hybrid estimate interpolates between the two scores, weighted by how well the two have agreed so far for that layer. Granularity is then chosen per layer. Let $k_\ell$ be the granularity used in layer $\ell$ and $r_\ell$ the fraction of weights removed there. Let $\hat\Delta_\ell(k_\ell, r_\ell)$ be the rescored real-loss increase, and let $T_\ell(k_\ell, r_\ell)$ be the latency of that layer measured with a lookup table profiled on the target device, whether a mobile CPU, an Ampere GPU with sparse tensor cores, or an NPU. We solve
$$\min_{\{k_\ell, r_\ell\}} \sum_\ell \hat\Delta_\ell(k_\ell, r_\ell) \quad \text{s.t.} \quad \sum_\ell T_\ell(k_\ell, r_\ell) \le T_{\text{budget}}.$$
The objective is additive over layers, so a greedy or small dynamic-programming solver suffices. After pruning, the network is fine-tuned through the proxy with SR-STE, and a final short round of real-codec rescoring checks that fine-tuning did not reopen the gap.
Baselines
The naive reference is the unpruned High Complexity wrapper run through the real codec, together with plain HEVC (or BPG) with no wrapper. For the sandwich HEVC tasks, the numbers to preserve are the reported 6.5 dB PSNR gain for low-resolution transport and the 30% bitrate reduction at equal LPIPS from Sandwiched Video Compression. For the machine-vision testbed, the reference is the 20.3% bitrate saving for COCO detection over BPG reported by Preprocessing Enhanced Image Compression for Machine Vision. Any pruned model is judged by the fraction of these gains it keeps at a given measured latency.
The strongest competitor is a hand-narrowed dense wrapper, in the style of the Slim model from the sandwich papers. It is trained from scratch through the same proxy and matched to each latency budget on each device by uniform width scaling. It is the strongest competitor because dense small networks are always fast on every chip and need no sparse kernels. The research file notes that the Slim model has about 57k parameters, but no reported result found on Lacuna for the Slim wrapper's PSNR or bitrate relative to High Complexity, so we will retrain it ourselves. It gets the same data and the same number of training GPU-hours as our pruning plus fine-tuning. Beating it means a lower BD-rate at equal on-device latency, on at least two of the three devices.
The second competitor is uniform 2:4 sparsity trained with SR-STE through the proxy, following Zhou et al.. On ImageNet that paper reports 77.0% top-1 accuracy at 2:4 against a 77.3% dense ResNet-50, which shows the pattern can be nearly lossless in classification. We run it with the same fine-tuning budget. We also run DSA-style layer-wise channel allocation scored purely through the proxy, which represents the best existing budgeted pruner.
Two ablations isolate our ingredients. The first removes real-codec rescoring, so every decision uses proxy scores only, while keeping per-layer granularity. The second keeps rescoring but forces a single granularity across the whole network. If the first ablation matches the full method, our first claim is false in practice.
Why not just…?
Why not just train a small dense wrapper, as the Slim model already does? This is the real threat. On a mobile CPU, where 2:4 buys nothing, a narrow dense network may well be optimal. The matched-latency comparison above settles the question device by device. We expect pruning to win mainly at moderate compression ratios and on sparse-capable hardware. If the dense narrow model wins everywhere, that is a clean finding, and it would still tell practitioners to spend their effort on architecture search rather than pruning.
Why not just fix the proxy instead of working around it? A better proxy would shrink the problem, but the sandwich authors already use a fairly elaborate composite proxy and still report a gap. Our rescoring needs only forward access to the real codec, so it applies to any proxy. As a direct check, we will repeat the rank-correlation study with both the analytic virtual codec and a learned BPG proxy. If both show the same concentration of disagreement in code-shaping layers, the problem is structural and not a flaw of one proxy.
Why not just prune with proxy scores and let fine-tuning absorb the errors? Fine-tuning also runs through the proxy, so it cannot correct errors the proxy cannot see. It may even make them worse, by adapting the surviving weights to proxy artifacts. The proxy-only ablation, evaluated on the real codec after fine-tuning, measures exactly this. The mask-stability metric from Zhou et al. tells us whether proxy noise causes extra mask flickering.
Why not just distill the large wrapper into a small one? Distillation through the proxy inherits the same blind spot: the student learns to imitate the teacher's codes as the proxy sees them. We include distillation as an add-on to the dense narrow baseline, so that the comparison is not won merely by training signal.
Proposed Experiments and Decisive Checks
The decisive first experiment is the rank-correlation study. It is cheap, requiring forward passes only, and it decides whether the rest of the program is worth running. Its central readout is the per-layer Spearman correlation between proxy and real importance. The main experiment then traces BD-rate against measured latency on three devices, for the HEVC low-resolution transport task and the LPIPS perceptual task, and traces COCO detection mAP against bitrate for the BPG preprocessor. Every method is evaluated only through the real codec. We also report the post-pruning proxy gap, defined as proxy loss minus real loss, to test whether our method keeps it from growing as sparsity increases. We report temporal flicker on video as the frame-to-frame variance of the codes. This checks whether fine-grained sparsity in code-shaping layers worsens the known flickering problem.
The result that would most change our view is if proxy-only pruning matches real-codec rescoring at every latency budget. We would then conclude that the proxy gap, although real in absolute terms, is rank-preserving, which is a useful fact about these proxies in its own right.
References
- Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers
- Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers
- Preprocessing Enhanced Image Compression for Machine Vision
- A Preprocessing Framework for Video Machine Vision under Compression
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
- S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
- MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference
- A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations