Image quality models built on large vision-language models now agree closely with human raters. They are also slow, and the most accurate of them compare each test image against several reference "anchor" images, so their cost multiplies. The obvious remedy is 2:4 sparsity, a hardware-supported pattern in which two of every four consecutive weights are zero. Recent NVIDIA GPUs run this pattern at up to twice the dense math rate. Ordinary scattered ("unstructured") pruning gives no such speedup, because on these GPUs the cost of tracking and skipping zeros eats the savings. The difficulty is that 2:4 masks are rigid and hurt accuracy. The standard methods that choose them after training judge success by how faithfully each layer reproduces its original outputs.
Our central point is that a quality scorer does not need to reproduce its outputs. It only needs to keep them in the same order. If the pruned model rates image A above image B whenever the original did, any monotone rescaling of its scores is harmless. We therefore propose to fit the hardware-friendly sparse model using a pairwise ordering loss, computed on image pairs whose order is known for free, instead of a reconstruction loss. We will then test whether this extra freedom lets 2:4 sparsity keep the model's ranking accuracy while delivering measured wall-clock speedups.
Core Claim
Post-training 2:4 compression of a quality-scoring model should be fit to preserve pairwise order, not activations or absolute scores. The ordering loss relaxes the fitting problem: it ignores any monotone distortion of the output and puts the limited correction capacity on the pairs that are hard to tell apart. We expect this to recover most of the ranking accuracy that plain 2:4 pruning loses, at essentially the same wall-clock speedup, without retraining on small human-labelled datasets. The claim is specific enough to fail. If a pruned model fit to reproduce activations on quality images, or fit to match the dense model's scalar score, ranks images as well as the order-preserving version, the claim is wrong.
Where This Sits
Four papers supply the ingredients for the ranking side. RankIQA trains a Siamese ranking network on synthetic distortion ladders, where more distortion is guaranteed to mean lower quality. It then fine-tunes on small human rating sets and reports LIVE SROCC of 0.981. PieAPP shows that pairwise preference labels avoid the scale noise of absolute ratings. Q-Align turns mPLUG-Owl2 into a scorer by reading the probabilities it assigns to five rating words and taking their expected value. Compare2Score instead compares each query to anchor images and converts the soft preferences into a score, reaching SRCC 0.931 on KonIQ-10k.
We reuse three things from this work. The ladders give us free ordered pairs. Q-Align and Compare2Score are the models we compress. The motivating observation is that these models are judged only by rank correlation, yet no one has asked what that means for compressing them.
On the sparsity side, SR-STE shows that 2:4 networks trained from scratch nearly match dense accuracy (77.0% vs 77.3% top-1 for ResNet-50). ARMOR makes post-training 2:4 pruning less damaging by wrapping each sparse weight matrix between small block-diagonal matrices. It fits those wrappers by layer-wise reconstruction on 128 generic text samples. The result is a 1.57x layer speedup on Qwen 2.5-14B, against 1.86x for plain 2:4, at roughly 3–5% extra parameters.
We reuse ARMOR's parametrization but replace its objective. The prior work on unstructured overhead (Wiedemann et al., Grimaldi et al.) and the tensor-core kernel SMaT constrain the design. SMaT only beats dense cuBLAS above about 78% sparsity and needs reordering preprocessing. That points us to 2:4 as the deployment target at moderate sparsity. HALP warns that FLOP counts mislead on GPUs, so every speed claim here is a measured wall-clock number.
The closest ranking-meets-pruning works are SPvR and RankNAS. SPvR ranks filter groups by how often removing them flips classifications, then retrains from scratch. RankNAS uses a pairwise ranker to compare candidate architectures. In both, ranking is a tool for choosing structure in classifiers or translators. In our proposal, preserving the order of the pruned model's outputs is the fitting objective itself. We also avoid retraining, which RankIQA's experience suggests would overfit small rating sets.
Research Program
The target models are Q-Align (code) and Compare2Score, both built on mPLUG-Owl2. Every linear layer in the language model, and optionally in the vision encoder, is replaced by ARMOR's form $\hat W = A\,(W' \odot M)\,B$. Here $M$ is a 2:4 mask, $W'$ holds the retained weights, and $A$, $B$ are block-diagonal wrappers. The mask is first set by standard reconstruction, as in ARMOR. After that, the wrappers, and optionally the retained weights, are tuned end to end on ordered image pairs.
The pairs come from two sources, and neither needs new human labels. The first is RankIQA-style synthetic distortion ladders, whose order is guaranteed. The second is real images ordered by the dense model's own scores. The loss is a margin-weighted pairwise logistic loss:
$$ \mathcal{L} = \sum_{(i,j)} w_{ij}\,\log\!\left(1 + \exp\!\big(-\,\mathrm{sign}(s_i - s_j)\,(\hat s_i - \hat s_j)/\tau\big)\right), $$
In this expression, $s_i$ is the dense model's score for image $i$ (or the synthetic ladder rank), and $\hat s_i$ is the pruned model's score. The temperature $\tau$ sets the scale of the loss. The weight $w_{ij}$ grows as $|s_i - s_j|$ shrinks, which puts effort on close pairs.
Because the loss depends only on score differences, any global shift or monotone stretch of the pruned scores costs nothing. That slack is what we expect the wrappers to exploit. For Q-Align, $\hat s$ is the expected rating computed from the five rating-word probabilities. For Compare2Score, it is the MAP score computed from the anchor comparisons. For Compare2Score, the anchor side can be cached, so the 2:4 speedup applies directly to the per-anchor query passes that dominate its cost.
Two further analyses come along. First, a per-layer sensitivity study measures how often each layer, pruned alone, flips the model's pairwise order. The few most order-sensitive layers are then left dense, subject to a measured-latency budget, much as HALP allocates pruning against measured latency. Second, we break flip rates down by the gap between the two images' quality scores. This tests the hypothesis that reconstruction-fit pruning mainly damages fine distinctions between similar-quality images, which is exactly what rank correlations reward.
Baselines
The naive reference point is plain one-shot 2:4 magnitude pruning with no correction, run on the same models. It gives the ceiling on speed, because it adds no wrapper overhead, and the floor on accuracy. The uncompressed dense model gives the accuracy target. Its throughput is anchored by the figure reported in the Q-Align resource: 23 images per second on one RTX 3090.
The strongest existing method is ARMOR with its original objective, layer-wise reconstruction. We run it twice. Once uses its published generic-text calibration. The other uses task-matched calibration: the same images our method sees, with the same number of calibration samples and the same optimization steps. The task-matched version is the fairest and most dangerous competitor, because it separates "right data" from "right objective." The file gives no reported rank correlation for ARMOR on any quality benchmark; ARMOR reports language benchmarks only. The best reported numbers we can anchor to are its 1.57x layer speedup and the dense Compare2Score KonIQ-10k SRCC of 0.931. Our method beats task-matched ARMOR if it closes at least half of the remaining SRCC gap to dense on held-out datasets, at matched speed and matched wrapper size.
The ablation that isolates our key ingredient keeps everything identical: wrappers, masks, pair images, steps, and learning-rate sweep. It changes only the loss, replacing the pairwise ordering loss with mean-squared error to the dense model's scalar score. This is ordinary output distillation. If it matches the ordering loss, the benefit comes from end-to-end tuning, not from order preservation. A second ablation removes the close-pair weighting. A third removes the synthetic ladders, keeping only teacher-ordered real images.
Every method gets the same tuning budget, a small grid of learning rates and steps selected on a KonIQ-10k validation split. We report medians over three seeds.
Why not just…?
Why not just use task-matched reconstruction calibration, the way SparseGPT, Wanda, or ARMOR already work? This is the real threat. Reconstruction on quality images may already preserve the few output directions that carry the score. ARMOR shows that reconstruction with good wrappers roughly halves the perplexity gap on Llama-2-13B. Our argument is that reconstruction spends its capacity reproducing every hidden-state direction, most of which have no bearing on the five rating-word logits. We do not assume this is true; the head-to-head comparison with task-matched ARMOR decides it. If ARMOR ties us, we will report that order-preserving fitting adds nothing for these models, and the gap-stratified flip analysis will show where the errors occur either way.
Why not just distill into a smaller dense model or prune whole channels? Channel pruning shrinks matrices without special kernels. But HALP shows that GPU latency drops in steps and does not improve at all unless channel counts land on alignment boundaries. SPvR gets good accuracy only by retraining from scratch, which the small human rating sets in quality assessment cannot support. We include one HALP-style latency-matched channel-pruned model as an extra comparison. Distilling to a smaller dense model is a legitimate alternative we do not claim to beat. It needs a full training run, whereas our method is post-training.
Why not just quantize to 8-bit and keep absolute scoring? Quantization and 2:4 sparsity stack on the same hardware, so this is a complement, not a rival. We report 8-bit alone and 8-bit plus our 2:4 model. If 8-bit alone already reaches the target throughput with no rank loss, the case for sparsity weakens, and we will say so.
Why not just run unstructured pruning through cuSPARSE or SMaT? SMaT shows that cuSPARSE is up to 125x slower than SMaT on real matrices, and that even SMaT only beats dense cuBLAS above about 78% sparsity. We will run one unstructured model at that sparsity through SMaT on A100, for completeness.
Proposed Experiments
Calibration pairs come from KonIQ-10k training images plus synthetic ladders built on them. Evaluation is on the KonIQ-10k test split and, without further tuning, on LIVE, TID2013 and AGIQA-3K for cross-dataset transfer. The metrics are SRCC and PLCC against human ratings. We also report the rate of pairwise order flips relative to the dense model, broken down by quality gap. Speed is measured as end-to-end images per second, and per-layer speedup, on RTX 3090 and A100 with 2:4 sparse kernels.
The file gives no reported result on Lacuna for Q-Align's SRCC on these datasets. We will therefore measure the dense Q-Align numbers ourselves, and use Compare2Score's reported 0.931 on KonIQ-10k as the published anchor.
Three checks decide the proposal. First, at matched wrapper size and matched end-to-end speedup, the pairwise-loss model must beat both task-matched ARMOR and the mean-squared-error ablation on held-out SRCC, across seeds. Second, its advantage must concentrate on close-quality pairs, which confirms the mechanism. Third, for Compare2Score, the 2:4 model with cached anchors must reach a higher measured query throughput than the dense model while staying within 0.01 SRCC of it on KonIQ-10k.
If the first check fails, the result is still worth publishing. It would tell practitioners that ordinary reconstruction calibration is enough for compressing quality scorers, and it would show which layers can safely take 2:4 sparsity.
References
- RankIQA: Learning from Rankings for No-Reference Image Quality Assessment
- PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
- Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training
- Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity
- High Performance Unstructured SpMM Computation Using Tensor Cores
- HALP: Hardware-Aware Latency Pruning
- SPvR: Structured Pruning via Ranking
- RankNAS: Efficient Neural Architecture Search by Pairwise Ranking