Global Repair of Pruned Language Models with Nyström Tangent Kernels

When a large language model is pruned, meaning a fraction of its weights or channels is deleted to save memory and compute, its outputs drift away from those of the original model. Current repair methods fix this drift one layer at a time on a few hundred calibration sequences. They either rotate the surviving weights, re-fit them by least squares, or add small learned corrections to attention heads. Each method sees only its own layer's error. Because errors compound across dozens of layers, these local fixes break down beyond roughly 20–30% structured sparsity. We propose a one-shot repair step that instead corrects the gap at the model's output. It computes a single update to all surviving weights at once, chosen so that the linearized pruned model reproduces the dense model's outputs.

The obstacle is size. The exact computation requires the empirical neural tangent kernel, the matrix of inner products between the network's output gradients at different inputs, which tells you how a weight change at one input moves the output at another. Over a calibration set this matrix has billions of entries. Nyström approximation is the classic way to tame such a matrix: compare every point only against a small set of "landmark" points and reconstruct the rest from them. With landmarks, the repair becomes a kernel ridge regression with an $m \times m$ solve, where $m$ is a few hundred to a few thousand. That gives two things the field lacks: a cross-layer, retraining-free correction, and a cheap spectral quantity, the kernel's effective dimension, that we hypothesize predicts where the compression-accuracy cliff will occur for a given model and sparsity.

Core Claim

Restricting the repair update to the span of output gradients at a small set of landmark outputs gives three properties at once. The update is global across layers. It is regularized against calibration overfitting by its low rank. And it preserves any pruning mask, because gradients with respect to deleted weights are simply never formed.

Concretely, let $r \in \mathbb{R}^n$ be the gap between dense and pruned outputs over $n$ scalar output coordinates. These are token positions on the calibration set, each projected onto a random direction of the final hidden state. Let $J \in \mathbb{R}^{n \times p}$ be the Jacobian of these outputs with respect to the $p$ surviving weights, and let $J_Z \in \mathbb{R}^{m \times p}$ be its rows at $m$ landmark coordinates. We set the update to $\delta = J_Z^\top \alpha$ and solve

$$\alpha^\star = \arg\min_\alpha \; \lVert r - K_{nZ}\,\alpha \rVert^2 + \lambda\, \alpha^\top K_{ZZ}\, \alpha, \qquad K_{nZ} = J J_Z^\top,\; K_{ZZ} = J_Z J_Z^\top,$$

where $\lambda$ is a ridge strength. This is exactly Nyström kernel ridge regression on the tangent kernel. It needs $m$ backward passes to form $J_Z$, $m$ forward-mode passes over the calibration set to form $K_{nZ}$, and one $m \times m$ solve. We repeat it two or three times, re-linearizing around the updated weights, as in Gauss–Newton optimization.

The method borrows its machinery from Accelerated Linearized Laplace Approximation for Bayesian Deep Learning. That paper builds Nyström approximations of the tangent kernel from forward-mode products to scale Bayesian uncertainty estimates to ViT-B. It never changes the weights; we turn the same object into a weight correction. The statistical motivation for small $m$ comes from Optimal Rates of Sketched-regularized Algorithms for Least-Squares Regression over Hilbert Spaces. It shows that landmark counts proportional to the effective dimension keep the optimal error rate of the full estimator. That result bears directly on the calibration-overfitting problem.

Relation to Prior Work

Three post-training repair methods motivate the gap, and each is local.

  • RCPU frames structured-pruning repair as an orthogonal Procrustes problem per layer. It reports that unconstrained least squares overfits 128–512 calibration samples, and it degrades sharply above 30% sparsity. We keep its lesson, that regularization is essential, but replace a geometric constraint with a low-rank function-space one that accounts for how layers compound.
  • Restoring Pruned Large Language Models via Lost Component Compensation extracts singular directions of attention-output gaps and learns one scalar per direction. It skips feed-forward blocks because learning many components overfits. Our update touches attention and feed-forward weights alike, with rank controlled by $m$.
  • ARMOR wraps a 2:4 sparse core in block-diagonal correctors. These add roughly 3–5% overhead and cost speed (1.57x versus 1.86x for naive 2:4). Our update lives inside the existing mask, so it adds zero inference cost.

One related paper constrains rather than supplies our design. NTK-SAP uses the trace of the tangent kernel to choose what to prune before training. It reports 60.79% ImageNet top-1 accuracy at 95.6% sparsity. That shows the tangent spectrum carries pruning-relevant signal, but it uses a scalar proxy and never repairs a trained model. Landmark selection reuses ideas from Recursive Sampling for the Nystrom Method and Matrix Coherence and the Nystrom Method.

We are not proposing Nyström attention (Nyströmformer) or anchor-based KV caching (AnchorKV). Both approximate activations at run time. Neither touches weights.

Research Program

The first stage builds the solver for Llama-family 7B and 13B models. It starts from three kinds of pruned model:

  • unstructured 50% and 60% pruning,
  • 2:4 semi-structured pruning,
  • 20%, 30%, and 40% structured channel pruning.

Calibration uses 128 C4 sequences, matching the cited baselines. Output coordinates are random projections of final hidden states at every token. Landmarks are chosen either uniformly or by approximate leverage scores. The forward-mode passes needed for $K_{nZ}$ are batched across landmarks.

The second stage is analysis. We compute the effective dimension $d_\lambda = \mathrm{tr}\big(\tilde K (\tilde K + \lambda I)^{-1}\big)$ of the Nyström tangent kernel $\tilde K$ for each pruned model, where $d_\lambda$ counts the output directions the surviving weights can still move. We then test whether the fraction of the output gap $r$ that lies within those directions predicts post-repair accuracy across sparsity levels. If it does, practitioners get a cheap way to locate the performance cliff before running any benchmark.

The third stage probes a known risk. The linearization may fail exactly where repair matters most, because the function gap at 50%+ sparsity may be too large for a first-order model. We measure how far each Gauss–Newton step closes the true, nonlinear gap versus the predicted one, and treat a large mismatch as a reportable negative result.

Baselines

The naive reference is the pruned model with no repair, built with the same mask. For structured pruning this is WANDA-sp, which RCPU reports at about 16.70 WikiText-2 perplexity for Llama-7B at 20%. Every method shares this mask, so differences come only from repair.

The strongest structured-pruning competitor is RCPU, run with its released settings on our masks and calibration data. RCPU reports about 14.40 WikiText-2 perplexity for Llama-7B at 20% structured pruning, and 64.55% average zero-shot accuracy for Llama-2-13B at the same sparsity. Beating it means lower perplexity at 20%, and, more importantly, a clearly flatter degradation curve at 30–40%, where RCPU is known to break down.

For unstructured pruning the strongest competitor is RestoreLCC. It reports a "+2.01% mean accuracy gain over the best baseline" on LLaMA-7B at 50% on the commonsense suite. The file gives no absolute accuracy for it, so we rerun it on identical masks. Beating it means higher mean zero-shot accuracy at equal (zero) inference overhead.

For 2:4 pruning we compare against ARMOR. The file reports only relative figures: the perplexity gap reduced by nearly 50% on Llama-2-13B, and 39.51 on GPQA for Qwen 2.5-32B. No absolute WikiText-2 number for ARMOR was found on Lacuna. Beating it means matching its perplexity while keeping naive 2:4 speed.

The decisive baseline is calibration distillation. This is plain gradient descent on the KL divergence between dense and pruned outputs, applied to the same masked weights with the same data. Its wall-clock budget equals ours, and its learning rate and step count are tuned on a held-out C4 split.

Three ablations isolate the key ingredients:

  • a layer-local version of our solver, with the kernel built per layer, which isolates the cross-layer effect;
  • a random low-rank subspace of the same dimension $m$ instead of landmark gradients, which isolates the choice of subspace;
  • uniform versus leverage-score landmarks, which isolates landmark selection.

Why not just…?

Why not just re-fit each layer by least squares on the calibration set? RCPU evaluates exactly this. It finds that least squares warps representation norms and angles on 128–512 samples and loses to its rotation-constrained alternative. Our layer-local ablation is the regularized version of the same idea. If it matches the global solver, the cross-layer claim fails.

Why not just fine-tune with LoRA? RestoreLCC reports that LoRA, which searches a low-rank weight space blind to what pruning removed, often fails to converge at 60% sparsity. Unmerged, LoRA also adds latency; merged, it breaks sparse formats. Our update preserves the mask by construction. We still include LoRA, at matched compute, for completeness.

Why not just run a few hundred steps of distillation on the calibration data? This is the real threat. It optimizes the same global objective, has no linearization error, and may match us in-domain. Our bet is that the Nyström restriction acts as the right regularizer on tiny calibration sets, as the sketched-regression theory suggests. In that case the advantage should show up mainly out of domain: calibrate on C4, test on WikiText-2 and zero-shot tasks, and repeat with only 32 calibration sequences. If distillation matches us out of domain at equal compute, we will say so. The contribution would then shrink to the effective-dimension diagnostic, which distillation cannot provide.

Proposed Experiments

The main grid crosses Llama-7B, Llama-2-7B, and Llama-2-13B with the three pruning families and sparsity levels above. It reports WikiText-2 and C4 perplexity, and average accuracy on BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA, along with repair wall-clock time on one A100. All rows use 128 C4 calibration sequences, and every repair is rerun under one shared protocol.

Three checks decide the question. First, does global repair widen the margin over RCPU as structured sparsity grows from 20% to 40%? A widening margin is the signature of compounding errors being corrected. Second, does it beat matched-compute distillation out of domain with small calibration sets? Third, does the in-span fraction of the output gap rank sparsity levels and models by post-repair accuracy, with rank correlation clearly above a parameter-count baseline?

A smaller vision check tests whether the diagnostic generalizes. We prune ResNet-50 on ImageNet with Fisher-style channel scoring from Group Fisher Pruning for Practical Network Compression, which reports a 1.79x speedup with a 0.37% top-1 drop, and test the repair in place of its fine-tuning phase.