Repairing Pruned Models' Prediction Sets After the Fact, at 2:4 Speed

A pruned language model can keep its accuracy and still lose its reliability. When half of its weights are zeroed, its confidence scores flatten, so a conformal predictor has to return long lists of labels to keep its promise. A conformal predictor is a wrapper that outputs a set of candidate labels guaranteed to contain the true one, say, 90% of the time. Calibration-Preserving Pruning (CPP) documents this effect and fixes it by choosing which weights to remove using gradients of the conformal score. That fix costs over 8x the pruning time of standard Wanda, and it only works for unstructured sparsity, meaning zeros scattered at arbitrary positions. Unstructured sparsity gives no wall-clock speedup on GPUs at 50% because tracking the scattered nonzeros costs more than multiplying the zeros. The format GPUs actually accelerate is 2:4 sparsity, where every block of four weights keeps exactly two. Sparse Tensor Cores run this format at roughly 2x, but no one has measured what it does to prediction sets.

We propose to prune cheaply in the hardware-friendly 2:4 format and then repair the prediction sets with a post-hoc calibrator. The calibrator never changes the order of classes within an input. It is trained to make conformal sets small, and it is folded into the classifier head so that inference costs nothing extra. The argument rests on one structural observation. Rank-preserving calibration keeps the order of classes within each input. Conformal set size, however, depends on how scores compare across inputs, because a single threshold is shared by all of them. Pruning damage can therefore be split into two parts. Within-input damage means the true label slides down the model's own ranking, and only changes to the weights can undo it. Cross-input damage means the model loses track of which inputs are hard, and a cheap calibrator that varies per input can undo it. We give an oracle bound that measures the two parts separately, and we build the calibrator that targets the second.

Core Claim

Order-preserving calibrators are usually judged by Expected Calibration Error (ECE), the gap between stated confidence and observed accuracy. They are valued because they cannot change top-1 accuracy. Methods in this family include temperature scaling (dividing all logits by one learned scalar), Parameterized Temperature Scaling (PTS), which predicts a separate temperature for each input from its sorted logits, and the shared isotonic map of Mix-n-Match. The research file notes that their effect on conformal set size is uncharacterized.

Under the standard conformal score, a label enters the set if its calibrated softmax probability exceeds a threshold estimated on held-out data. The two kinds of transform behave differently. A single strictly increasing transform applied identically to every input's scores leaves the sets unchanged. A per-input temperature $T(x)>0$ also leaves each input's ranking unchanged, but it changes how inputs compare with one another, and that comparison is what sets the threshold.

Our claim is that most of the set-size inflation caused by fast 2:4 pruning is cross-input damage. If so, a per-input temperature trained directly for small sets should recover most of CPP's efficiency gain. It would do so without any gradient-based pruning, in a format that actually runs fast.

The claim is falsifiable through a bound. Let $r_i$ be the rank of the true label for test input $i$ under the pruned model's logits, and let $n$ be the number of test inputs. Any calibrator that preserves within-input order and thresholds its scores produces sets that are top-$k_i$ prefixes of that ranking. Reaching coverage $1-\alpha$ therefore requires $k_i \ge r_i$ on at least a $(1-\alpha)$ fraction of inputs. This gives

$$\bar{K}_{\min} \;\ge\; \frac{1}{n}\sum_{i \in \mathcal{S}} r_{i}, \qquad \mathcal{S} = \text{the } \lceil (1-\alpha) n \rceil \text{ inputs with smallest } r_i,$$

where $\bar{K}_{\min}$ is the smallest mean set size any order-preserving recalibration of this pruned model could achieve.

The bound separates the two kinds of damage. The difference between the bound for the dense model and the bound for the pruned model measures within-input damage, which only mask surgery such as CPP can fix. The difference between the set size actually achieved and the pruned model's bound measures cross-input damage, which is the room a post-hoc calibrator can exploit. If the second difference is small, our proposal fails, and that failure would itself be evidence that CPP-style pruning-time intervention is necessary.

Research Program

The work proceeds in three stages.

The first stage is a diagnostic. Following CPP's four-way split protocol (pruning, validation, conformal calibration, test), we prune Qwen2.5-1.5B, Llama-3-8B, and RoBERTa-base on CPP's classification suite at 50% sparsity. We use both the unstructured and the 2:4 format, with Wanda and SparseGPT as pruners. We then compute the prefix bound and the achieved set size before and after recalibration. The 2:4 measurements are new: no reported result was found on Lacuna for conformal set size of 2:4 pruned models.

The second stage builds the calibrator. It is a per-input temperature, $T(x)=\mathrm{softplus}(w^\top h(x)+b)+\tau_0$, where $h(x)$ is the final hidden state, $w$ and $b$ are learned weights and bias, and $\tau_0>0$ is a fixed floor that keeps the temperature positive. This draws on PTS, but PTS reads only the top-10 sorted logits. The research file flags that restriction as a bottleneck, and we lift it by reading the internal representation. Because $w^\top h$ is just one more output row, the temperature head is appended to the classifier or label-word projection as an extra column and runs inside the same matrix multiply.

The calibrator is trained on the validation split with a smooth surrogate for set size. The surrogate softens the threshold-inclusion indicator with a sigmoid, and a cross-entropy anchor is added so that ECE does not degrade. Conformal calibration is then run on its own disjoint split, which keeps the coverage guarantee exact.

The third stage asks whether the 2:4 format and the calibrator compose with stronger 2:4 pruners. We substitute ARMOR for the pruner. ARMOR wraps a 2:4 sparse core between small block-diagonal matrices and recovers up to half of the perplexity lost to 2:4 pruning. We test whether its rotations change how much cross-input damage remains.

Baselines

The naive reference is a 2:4 Wanda or SparseGPT model with plain split conformal and no recalibration. A second reference adds a single global temperature tuned by negative log-likelihood on the validation split. Global temperature scaling is free at inference, since the scalar folds into the head, so it is the cheapest competitor and the first thing a practitioner would try.

The strongest existing method is threshold-aware CPP, the only method in the file that targets conformal efficiency under pruning. On its home setting it reports a mean set size of 8.6 labels on DBpedia-14 at 90% coverage and 50% unstructured sparsity, against 10.1 for standard SparseGPT (CPP). We run it in two ways. The first reproduces it unstructured. The second ports it to 2:4 by using CPP saliency to choose which two weights survive in each group of four. CPP itself never formulated this variant, so it is the obvious idea at this intersection, and it is the one we must beat.

The comparison is held fair as follows. All methods use identical splits and identical pruning data. Every method's calibrator or saliency weight $\lambda$ gets the same validation budget for hyperparameter search. We report offline pruning time, measured on the same A100, alongside quality.

We count our method as beating CPP under two conditions. First, 2:4 pruning plus our calibrator must match or beat 2:4-CPP's set size at equal coverage in a majority of dataset and model configurations, at a fraction of its pruning cost. Second, on the unstructured setting, it must come within the 10.1-to-8.6 gap on DBpedia-14. A stronger outcome would be that our calibrator on top of 2:4-CPP improves further, which would show the two kinds of damage are complementary.

Three ablations isolate the key ingredient. The first swaps the hidden-state features for PTS's sorted top-10 logits. The second trains with negative log-likelihood instead of the set-size surrogate. The third compares a global temperature with a per-input one. Together they distinguish whether the gains come from representation features, from the objective, or from cross-input re-ranking itself.

Why not just…?

Why not just apply global temperature scaling after 2:4 pruning? A single scalar does change sets through softmax renormalization, but it has one degree of freedom and cannot tell easy inputs from hard ones. PTS reports an average 30% relative ECE reduction over Ensemble Temperature Scaling precisely because difficulty varies across inputs (PTS). That result concerns ECE, not set size, so the global-temperature baseline is a genuine test. If it closes most of the gap to the prefix bound, our calibrator is unnecessary, and we will report that.

Why not just port CPP's saliency into the 2:4 mask? This is the most serious alternative. It may beat us wherever the diagnostic shows large within-input damage, since no order-preserving calibrator can move the true label up the ranking. Its cost is the 8x offline overhead CPP reports. The prefix bound tells us in advance on which datasets it should win. We run it head-to-head and also stacked with our calibrator.

Why not just use a better 2:4 pruner such as ARMOR? Better reconstruction might restore the model's score geometry without any calibration. The third stage of the program tests this directly: if ARMOR's prefix bound and achieved set size match the dense model's, the calibrator adds nothing there. ARMOR's own speedup, 1.57x on a Qwen 2.5-14B layer in unoptimized PyTorch (ARMOR), also gives us a realistic latency budget to compare against.

Why not just run unstructured CPP models through reordering kernels like SMaT? The file reports that cuSPARSE-style libraries are inefficient at 50 to 90% sparsity (Accelerating Sparse Neural Networks on Dense Hardware). We will nonetheless include a measured latency row for unstructured CPP under the best available sparse kernel, so that the speed argument is empirical rather than assumed.

Proposed Experiments

The decisive check comes first. We compute the prefix bound and achieved set size for dense, 2:4, and unstructured models on DBpedia-14, Banking77, and CLINC150, the large-label sets where CPP saw the largest effects. AG News and TREC serve as small-label controls. The central prediction is that achieved set size under 2:4 sits well above the pruned bound, so cross-input damage dominates, and that our calibrator closes most of that difference. Coverage must remain within finite-sample tolerance of 90%.

Secondary metrics are top-1 accuracy, which should be unchanged by construction, and kernel-density ECE from Mix-n-Match, used because binned ECE is noisy. We also report WikiText-2 perplexity as a sanity check.

Systems measurements on an A100 compare end-to-end latency in three configurations: dense, 2:4 with the fused temperature column, and 2:4 with PTS as a separate network. These test whether fusing the calibrator truly removes its overhead. We also report pruning wall-time against CPP.

Two outcomes would be decisive. If our method beats 2:4-CPP at a fraction of its pruning cost, the claim holds. If 2:4-CPP wins exactly where the bound predicts large within-input damage, the decomposition itself is validated, which tells practitioners when gradient-based pruning is worth its cost.