Perceptually Weighted 2:4 Pruning for Networks That Output Color

Networks that correct or restore color, such as white-balance models, are judged by people. Yet when they are compressed, the pruning step measures damage with plain RGB arithmetic. Color science has long established that equal steps in RGB do not look equal. A shift that is invisible in saturated greens is obvious in a neutral gray. A shift that crosses a named-color boundary, say from blue to purple, looks larger than a same-sized shift inside one category (Image Quality Assessment and Color Difference). Pruning on modern GPUs is also tightly constrained. To get real speedups, weights must follow a rigid pattern such as 2:4 sparsity, which zeroes exactly two of every four consecutive weights so that Ampere sparse Tensor Cores can skip them. Under such a constraint, every layer must discard some weights it would rather keep. The real question is where in the output the resulting error lands.

We propose to make that choice perceptually. During one-shot, post-training 2:4 compression, the per-layer reconstruction objective is weighted by how much each image location's error would cost under a perceptual color difference (CIEDE2000, the standard formula for perceived color difference), pulled back from the network's output to the layer being pruned. The claim is concrete and testable. For a color-to-color network at a fixed hardware-friendly sparsity pattern and fixed calibration data, a perceptually weighted mask should give lower ΔE2000 and fewer color-category flips than masks chosen to minimize RGB error. It should do so at identical latency, because only the choice of zeros changes, not their structure.

Why this gap, and why not the obvious one

The research file points to an apparently obvious bridge: converting the RGB input to an opponent color space so that the first layer, which Channel Permutations for N:M Sparsity explicitly cannot permute, becomes prunable. We reject it. That paper itself notes the input layer is tiny (three channels) and is rarely pruned. A linear RGB-to-opponent transform also folds directly into the first convolution's weights, so it barely changes the optimization problem. Speeding up a negligible layer is not worth a paper.

The real misalignment sits elsewhere. It lies in the objective that decides which weights die in all the heavy layers.

Two bodies of work motivate this. The sparsity side has learned that unstructured pruning rarely pays off on real hardware. Sparse GPU Kernels for Deep Learning shows generic sparse kernels lose to dense libraries until very high sparsity, even though its Sputnik kernels beat dense cuBLAS above roughly 71%. For this reason practitioners adopt 2:4 patterns (Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch reports a 2x hardware-supported speedup) or tile-aligned patterns (Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity).

These rigid patterns cost accuracy precisely because the pattern, not importance, forces some salient weights out. That cost is worst in small networks, as the 4.01% SqueezeNet drop in the channel-permutation paper shows. The color side shows that the human cost of an error depends strongly on where it falls in color space. PerSIM (PerSIM) separates luminance from chroma in CIELAB and weights them asymmetrically. PCDM scores category-crossing shifts separately from fine shifts.

ABC-Former shows that a strong white-balance network benefits from explicit CIELAB conditioning. Its authors also flag that the model is too heavy for edge deployment, and they offer no compression strategy. None of these works lets the perceptual metric choose the sparsity mask, and none of the pruning works knows that its outputs are colors.

Core Claim

Take a layer with weights $W$ and calibration inputs $X$, where each column $x_t$ corresponds to one spatial token $t$. One-shot pruners in the SparseGPT and Thanos family choose a structured mask and updated weights $\hat W$ that minimize the unweighted error $\sum_t \|(W-\hat W)x_t\|^2$. We instead minimize

$$\sum_t d_t \,\|(W-\hat W)x_t\|^2, \qquad d_t = \mathbb{E}\big[\|J_t^{\top} \nabla_{o}\, \Delta E_{00}(o, o^{*})\|^2\big],$$

where $o$ is the network's sRGB output, $o^*$ the target, $\Delta E_{00}$ the CIEDE2000 difference, and $J_t$ the Jacobian of the output with respect to the layer's activation at token $t$. In words, $d_t$ estimates how much a unit of error at token $t$ in this layer would move perceived color at the output. It is a perceptual Fisher-style weight, estimated with one backward pass per calibration image. The weighted problem keeps the same quadratic structure, with Hessian $X D X^{\top}$ and $D=\mathrm{diag}(d_t)$. Existing 2:4 and 4:8 solvers therefore run unchanged.

We expect three things to follow. Error mass will move away from neutral and near-boundary pixels toward saturated, perceptually tolerant regions. Mask choice will matter most where fine-tuning budget is small. And the gain over plain CIELAB-space calibration will concentrate in exactly the regions where CIEDE2000 corrects CIELAB, namely neutrals and blues.

Research Program

We first build the weighted solver inside the Thanos codebase. It already implements Magnitude, Wanda, SparseGPT and Thanos for unstructured, 2:4 and 4:8 patterns, so our change is confined to the Hessian accumulation.

The subject network is an ABC-Former-style white-balance model trained on Rendered WB. We use released weights if they exist; otherwise we retrain from the paper's description and verify that we reach its reported range before compressing. As a second, convolutional subject we use a small U-Net white-balance model. Small networks are where rigid patterns hurt most, according to the channel-permutation results.

Second, we analyze where the error goes. For each pruned model we map per-pixel ΔE2000 against the target's position in color space (lightness, chroma, hue bins) and against PCDM-style 11-category membership. This lets us test whether RGB-calibrated masks really push errors into grays and category boundaries.

Third, we test whether the benefit survives the fine-tuning that practitioners would add. We sweep a sparse fine-tuning budget from zero steps to a full schedule using S-STE (S-STE), which reports more stable 2:4 training than SR-STE (78.5% versus 75.7% on DeiT-small).

Fourth, we check that the gains appear at real speedups. We measure end-to-end latency for 2:4 on an Ampere GPU, tile-wise sparsity on CUDA cores, and unstructured 75–90% sparsity through Sputnik. Perceptual weighting does not change any mask's structure, so latency should match the RGB-calibrated counterpart exactly. Any deviation would point to a measurement bug.

Baselines

The naive reference points are two. The first is the dense model, which sets the quality ceiling; on Rendered WB Set1-Test, ABC-Former reports an MSE of 20.47 (ABC-Former). The paper says it achieves the lowest ΔE2000 but gives no number in the file, so for ΔE2000 there is no reported result found on Lacuna for ABC-Former. We will measure the dense ΔE2000 ourselves. The second naive reference is one-shot magnitude 2:4 pruning with no calibration, which is the floor.

The strongest existing one-shot competitor is second-order 2:4 pruning with RGB-MSE calibration: SparseGPT and Thanos as implemented in the Thanos codebase, plus Wanda. These solve the same quadratic problem we do, only unweighted, so they isolate our change. They will use the same calibration images, the same number of calibration samples, the same block sizes and damping. Any hyperparameter search granted to our method will be granted to them as well. No reported result was found on Lacuna for any of these pruners on white-balance or other color-output tasks; the closest structured-sparsity numbers are classification results, such as SR-STE's 77.0% versus 77.3% dense ResNet-50 top-1. Beating these baselines means a statistically reliable reduction in mean ΔE2000 and in category-flip rate on Set1-Test, Set2 and Cube+, at matched sparsity pattern, with RGB MSE no more than marginally worse.

The strongest competitor when training is allowed is S-STE sparse fine-tuning with a ΔE2000 loss, starting from the RGB-calibrated mask. To compare it with S-STE started from our mask, both runs get identical step budgets.

Two ablations isolate the key ingredient. In the first, $d_t$ is replaced by a pixel-independent luminance/chroma weighting (a fixed 3×3 opponent reweighting). This asks whether the per-pixel, color-dependent nature of the metric matters, or merely a global emphasis on luminance. The second is calibration with L2 in CIELAB rather than CIEDE2000, which we treat as the main threat (see below). Channel permutation (Channel Permutations for N:M Sparsity) is orthogonal, so we report it both with and without our weighting instead of as a rival.

Why not just…?

Why not just fine-tune the pruned model with a ΔE2000 loss? This is the most serious threat. Given enough steps, a fine-tuned model may recover regardless of the initial mask, especially since 2:4 training methods like SR-STE and S-STE let masks move during training. Our claim is therefore scoped to the low-budget regime that edge deployment actually faces. The fine-tuning-budget sweep settles it directly. If the curves for the two starting masks converge after a few hundred steps, we will say the method only matters for zero-shot compression. If they stay separated, mask choice carries information that fine-tuning does not recover.

Why not just convert the outputs to CIELAB and calibrate with ordinary L2 there? CIELAB L2 is the older ΔE76 formula. It already captures much of the luminance/chroma asymmetry that PerSIM exploits, and it costs nothing extra. If it matches our method, the honest conclusion is "calibrate in CIELAB," and that is still a useful, publishable finding. We expect CIELAB L2 to fall short specifically in neutrals and blues, where CIEDE2000 was introduced to fix CIELAB's non-uniformity. The binned error analysis tests exactly this. A gain that appears only in aggregate and not in those bins would undercut our explanation.

Why not just run unstructured sparsity with a good kernel, or keep the model dense and smaller? Sputnik and SMaT (High Performance Unstructured SpMM Computation Using Tensor Cores) only beat dense cuBLAS above roughly 71–78% sparsity, and SMaT's offline reordering adds preprocessing cost. A white-balance network whose quality hinges on fine color may not tolerate that sparsity. We include unstructured 75–90% runs through Sputnik, weighted and unweighted, and a dense width-reduced model at matched latency. This tells us whether structured-but-perceptual sparsity sits on a better quality–latency frontier than either option.

Why not just keep the color-sensitive layers dense? Layer-wise allocation is cheap and may capture much of the benefit. We test it as a variant, using $d_t$ summed per layer to rank layers. We then report whether token-level weighting adds anything once the layer allocation is perceptual.

Decisive Checks

The method is worth pursuing if three things hold. At zero fine-tuning steps and 2:4 sparsity, perceptual calibration lowers mean ΔE2000 and category-flip rate relative to Thanos/SparseGPT with RGB calibration on all three Rendered WB / Cube+ splits. The gain exceeds that of CIELAB-L2 calibration, and the excess concentrates in neutral and blue bins. And latency matches the RGB-calibrated mask exactly. As an independent check that the improvement is not an artifact of optimizing the evaluation metric, we will also score outputs with PerSIM and PCDM, which were not used during pruning. The method is falsified if the pixel-independent opponent weighting ablation matches it, or if CIELAB-L2 calibration matches it bin by bin.