Pruning Speech Rescorers for What They Compare, Not What They Compute

Speech recognizers often run in two passes. A fast first pass proposes a short list of candidate transcripts, typically 10 to 100 near-identical sentences. A large neural language model then rescores that list and picks the best one. The second-pass model never needs to say how good a sentence is in absolute terms. It only needs to rank candidates that differ in one or two words, such as "their" versus "there," or a dropped article. We propose compressing these rescorers with a one-shot pruning method that keeps only what matters for that comparison. It preserves the model's behavior on the differences between competing hypotheses and treats everything the hypotheses share as expendable.

This matters because the two relevant literatures point in opposite directions. Rescoring accuracy keeps improving as rescorers grow from 5M to 700M parameters, but latency budgets in streaming speech stay at tens of milliseconds. Separately, large language models turn out to be highly redundant and can lose half or more of their weights in one shot with little damage. The standard one-shot pruners, however, try to reproduce every layer's full output on generic text. That spends the pruning budget on the part of the computation that cancels out when candidates are compared. If the idea works, a pruned 700M-parameter rescorer would beat a dense or distilled model with the same number of nonzero weights. It would also tell us whether rescorer scaling laws track the parameters a model was trained with or the parameters it keeps.

Core Claim

A discriminative rescorer is trained with minimum word error rate (MWER), a loss that minimizes the expected edit distance over the candidate list. Its output enters that loss only through a softmax over the scores within one utterance's list. Adding any constant to every candidate's score leaves both the loss and the chosen transcript unchanged. So the part of each layer's input that is common to all hypotheses of an utterance matters only indirectly, while the part that varies across hypotheses determines the ranking.

SparseGPT (SparseGPT) prunes each layer by solving a sparse regression. For a layer with weights $W$ and calibration inputs $X$, it seeks sparse $\hat W$ minimizing $\|WX - \hat W X\|^2$, and it uses the input second-moment matrix $H = XX^\top$ to decide which weights to remove and how to compensate the survivors. We replace this with a contrastive version:

$$ H_\Delta = \sum_{u}\sum_{i} p_{u,i}\,(x_{u,i} - \bar{x}_{u})(x_{u,i} - \bar{x}_{u})^\top + \lambda\, XX^\top . $$

Each symbol has a specific meaning. Here $x_{u,i}$ is the layer input at a token position of hypothesis $i$ for utterance $u$, and $\bar{x}_u$ is the posterior-weighted mean input at the same position across that utterance's hypotheses. Positions are matched across hypotheses using a Levenshtein alignment to the 1-best, and tokens with no counterpart are left uncentered. The weight $p_{u,i}$ is the current rescorer's posterior for hypothesis $i$. The small constant $\lambda$ keeps the common-mode signal from being ignored entirely, which also keeps the matrix well-conditioned.

The pruning solver is otherwise SparseGPT's, so we are reusing its method, not replacing it. The new part is what counts as "preserving the layer": we reconstruct the between-hypothesis variation that drives the ranking, weighted toward the hypotheses the model actually considers plausible. We call this margin-preserving pruning.

The claim is testable in three ways. At equal sparsity with no retraining, margin-preserving pruning should keep more of the dense rescorer's WER gain than SparseGPT calibrated on the same N-best text. The advantage should grow with sparsity. And a large pruned rescorer should sit above the dense scaling curve when accuracy is plotted against nonzero parameters.

Where This Sits in Prior Work

Three works motivate the question by showing that rescorer quality depends on scale and on the discriminative objective. Discriminative Speech Recognition Rescoring with Pre-trained Language Models shows that MWER-trained BERT rescorers beat likelihood scoring. With Whisper-tiny candidates, test-clean WER falls from 5.67% to 4.31%. The same paper shows that a pooled scalar head (RescoreBERT) cuts latency from 6115 ms to 33 ms by avoiding one forward pass per masked token. Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems finds that bidirectional models help where causal GPT-2 does not, with the gains concentrated on low-frequency words. Those rare-word distinctions are exactly the fine differences a generic pruner might sacrifice. Scaling Laws for Discriminative Speech Recognition Rescoring Models fits power laws for dense pretrained rescorers from 5M to 700M parameters, finding that a 10× increase in parameters is worth about 5.3× more data. That work never asks whether the relevant variable is parameters at training time or parameters retained afterwards. Our scaling analysis extends it directly.

Two works on compressing rescorers constrain the design, because each is the obvious alternative. Low-Rank Adaptation of Large Language Model Rescoring (LoRB) adapts rescorers up to 1B parameters by training small low-rank add-on matrices. It matches full fine-tuning on test-other (10.78% vs 10.80%), but the model stays fully dense at inference, so LoRB answers a training-cost question rather than ours. Distillation Strategies for Discriminative Speech Recognition Rescoring trains a 5M dense student from a 170M teacher, and the student recovers all of the teacher's gains on production data. That is a serious threat. We address it head on below by moving to larger teachers, where more gain is at stake, and by comparing at matched budgets.

The redundancy literature supplies both the method and the hypothesis. SparseGPT supplies the solver. It also shows that models above roughly 10B parameters survive 50–60% one-shot sparsity, while models under 1B suffer steep drops. That size limit is a negative result our proposal must confront, since rescorers are 5M–1B. We argue that the discriminative objective changes the picture: measured only on the between-hypothesis subspace, a small rescorer may be far more redundant than its full-output reconstruction error suggests. Universal Properties of Activation Sparsity shows that the effective rank of activations falls as models grow, which supports the idea that the useful signal occupies a small subspace. Two works give the hypothesis for the matched-budget comparison: To Prune, or Not to Prune finds that large sparse models beat small dense ones, and Double Sparse Factorization finds that a 50%-pruned 13B model beats a dense 7B. None of these works evaluates a model trained to rank lists, and none uses a pruning objective that exploits the fact that the rescorer's output is invariant to a common shift across candidates.

Research Program

We first build a family of MWER-trained pooled rescorers on LibriSpeech N-best lists. The first pass is Whisper-tiny, following Shivakumar et al., with 16-best and 100-best lists. The rescorers span the pretrained encoder sizes we can obtain, targeting roughly 5M to about 1B parameters so that the range covers the 5M–700M span of the scaling-law study.

For each model, we measure how redundant it is in two ways. The first is the effective rank of its layer inputs, computed as usual. The second is the effective rank of the aligned, hypothesis-centered differences. We predict the difference subspace has much lower rank than the full activation space, and that the gap widens with model size. If it does not, the core mechanism has little to exploit, and we learn that early.

We then prune each rescorer three ways at 50–90% unstructured sparsity and at 2:4 semi-structured sparsity. The first two are magnitude pruning and SparseGPT with the standard matrix $XX^\top$, both calibrated on the same N-best transcripts. The third is margin-preserving pruning with $H_\Delta$. We do this with no retraining, and again with a short MWER healing phase at fixed budgets of 0, 1%, and 5% of the original fine-tuning steps.

For latency on real hardware, we also test a structured variant of the idea. We apply ReplaceMe-style layer removal (ReplaceMe), which cuts blocks and fits a linear shortcut, but we choose layers by how little they change the between-hypothesis differences, rather than by ReplaceMe's cosine distance on raw activations.

Finally, we plot $WER_{norm}$ against nonzero parameters for all dense, distilled, and pruned models. $WER_{norm}$ is the fraction of the gap between first-pass WER and oracle WER that the rescorer closes. The plot tests whether pruned large models trace the dense power law of Gu et al. or lie above it.

Baselines

The naive reference points are the first pass alone and the dense rescorer. The first pass alone, Whisper-tiny's 1-best, is reported at 5.67% test-clean WER in Discriminative Speech Recognition Rescoring with Pre-trained Language Models. The uncompressed MWER rescorer, reported at 4.31% test-clean in the same paper, is the ceiling. Every compressed model is scored by the fraction of this 1.36-point gap it retains. Beating a baseline means retaining a larger fraction at the same nonzero-parameter count and the same measured latency.

The strongest existing competitor is discriminative distillation into a dense student, following Distillation Strategies for Discriminative Speech Recognition Rescoring. That paper reports that a 5M student recovered 100% of a 170M teacher's gains on production General and Tail domains. We found no reported LibriSpeech number for this distilled student on Lacuna. We will reproduce it using the paper's post-cross-entropy and MSE objectives. For fairness, the student is trained from the same teacher, on the same N-best lists, and its nonzero-parameter count is matched to each pruned model. It gets a tuning budget at least as large as the pruned models' healing budget, measured in GPU-hours. We explicitly allow distillation more compute than the one-shot methods. The pruned model beats distillation if it retains more of the teacher's gain at matched size, with the difference larger than the spread across three random seeds.

The ablation that isolates our key ingredient is SparseGPT with its standard matrix $XX^\top$, calibrated on exactly the same N-best hypotheses and posteriors. The only difference is centering and posterior weighting. This separates "calibrate on in-domain N-best text" from "reconstruct only the differences." We add a second ablation that keeps the centering but drops the posterior weights $p_{u,i}$, to see whether focusing on plausible hypotheses matters. Magnitude pruning is a sanity floor. Its reported collapse beyond 30% sparsity on large models in SparseGPT tells us to expect it to fail early.

We also report 8-bit and 4-bit quantization of the dense rescorer, alone and combined with each pruning method. The goal is to show whether the pruning gains stack with quantization, as SparseGPT found for perplexity.

Why not just…?

Why not just distill into a small dense model, since it already recovers all the teacher's gains? This is the real threat. At 170M→5M, as reported in Distillation Strategies, there may be nothing left to win. We therefore move the comparison to teachers of 700M–1B parameters, where Gu et al.'s scaling law implies more gain is at stake and a 5M–70M student has further to fall. We compare at every matched budget rather than only the smallest one. If distillation matches pruning everywhere along the curve, we report that as the answer. A secondary benefit of pruning would still stand: a pruned model needs only calibration data and minutes of compute, not a full distillation training run. But we will not claim accuracy we do not observe.

Why not just run standard SparseGPT on N-best text and heal with a little MWER fine-tuning? This could plausibly close the gap, which is why healing budgets are an explicit axis. If margin-preserving pruning wins only at zero healing and the advantage vanishes after 1% of fine-tuning steps, then the contribution shrinks to cheaper deployment. We will say so. The stronger outcome we are testing for is that a better starting point stays ahead at high sparsity (above 70%), where healing must rebuild what the pruner destroyed rather than nudge it.

Why not just quantize? Quantization reduces the bits per weight but does not remove weights or compute. SparseGPT reports that 50% sparsity combined with 4-bit quantization beats 3-bit quantization at the same memory. Our combined runs test whether the same holds for WER. If quantization alone meets the latency target at no WER cost for the model sizes that matter in streaming, the case for pruning rests only on the larger models, and we will report where that crossover falls.

Why not just use a smaller pretrained dense rescorer trained with MWER? Gu et al.'s power law already predicts how such a model performs. It is the dense curve in our scaling plot, so the plot answers this question directly.

Proposed Experiments and Decisive Checks

The main benchmark is LibriSpeech test-clean and test-other with Whisper-tiny 16-best and 100-best lists. We report WER, $WER_{norm}$, nonzero parameters, and measured latency per list. Latency is measured on GPU for 2:4 and depth-pruned models and on CPU with a sparse inference engine for unstructured models, since SparseGPT reports a 2.16× CPU speedup with such an engine. As a second domain with harder, rare-word-heavy errors, we use Switchboard and CallHome, where Udagawa et al. report absolute gains of up to 0.3 and 0.5 on 16-best lists. We break errors down by word frequency and by error type (insertions, deletions, substitutions). This tests the specific worry that generic pruning erases rare-word discrimination.

Four checks decide the project. First, at 70% sparsity with no healing, margin-preserving pruning must retain clearly more of the dense rescorer's gain than same-data SparseGPT, and the advantage must hold across model sizes. Second, the rare-word insertion errors that bidirectional rescorers fix should be where standard pruning loses most and our method loses least. If the losses are spread uniformly, the mechanism story is wrong even if the averages favor us. Third, on the $WER_{norm}$-versus-nonzero-parameters plot, pruned large models must lie above both the dense curve and the distilled students for the claim "effective parameters, not trained parameters" to hold. Fourth, the between-hypothesis differences must have much lower effective rank than the full activations. If they do not, any win must come from something other than the proposed mechanism, and we will look for it.