Deep neural networks are notoriously redundant. When compressing these models via structural pruning—removing entire neurons, channels, or attention heads—the traditional pipeline involves dropping the parameters and then running expensive, gradient-based fine-tuning to recover the lost accuracy.
However, a powerful alternative paradigm has emerged: least-squares weight reconstruction. Instead of treating the post-pruning recovery as a non-linear optimization problem solved by backpropagation, we can treat it as a localized, linear signal-reconstruction problem. Because a layer's output is a linear combination of its weights and inputs, we can compute a closed-form mathematical adjustment to the remaining parameters that directly minimizes the distortion of the layer's output activations.
The Core Intuition: Pruning as a Signal Recovery Problem
To understand why this works, consider a single layer in a neural network. The layer takes an input matrix $X$ and multiplies it by a weight matrix $W$ to produce an output feature map $Y = XW$.
When we prune a subset of neurons or channels, we are effectively dropping columns or rows from $W$, resulting in a smaller weight matrix $W_{\text{pruned}}$ and a distorted output $\hat{Y} = XW_{\text{pruned}}$. The standard approach accepts this distortion and relies on downstream layers and gradient descent to adapt over many epochs.
Least-squares reconstruction asks a simpler question: Can we adjust the remaining weights in $W_{\text{pruned}}$ such that $\hat{Y}$ is as close to $Y$ as possible?
Mathematically, we want to minimize the reconstruction error:
$$\min \| Y - XW_{\text{pruned}} \|_2^2$$
Because this objective is quadratic with respect to the weights, it has a well-known analytical solution via the ordinary least squares formula. By solving this linear system, we can instantly compensate for the dropped parameters without running a single step of backpropagation.
As Park et al. (2024) demonstrate, solving this linear least-squares problem layer-by-layer allows large language models to recover their performance at a fraction of the cost of gradient-based retraining. This insight proves that expensive backpropagation is not strictly necessary for restoring model performance after structural changes. In fact, even when channels are selected completely at random, Li et al. (2022) showed that a least-squares optimization step to align feature maps can instantly preserve the network's representational capacity before any fine-tuning even begins.
Feature Map Alignment
The primary reason this approach works so well is that deep neural networks are highly sensitive to changes in their intermediate feature maps. As Zhang et al. (2017) and Yang et al. (2017) established, minimizing the reconstruction error of the output feature maps preserves the functional behavior of a layer far better than trying to minimize the change in individual filter weights. By framing pruning as a feature map alignment problem, the remaining channels absorb the responsibilities of the discarded ones.
This closed-form alignment is so reliable that it can serve as a fast proxy for evaluating sub-network quality. For instance, Li et al. (2022) utilized least-squares weight reconstruction within an evolutionary algorithm framework to prune vision transformers and CNNs. Instead of fine-tuning thousands of candidate architectures during the evolutionary search—which would be computationally prohibitive—the authors used the closed-form reconstruction to instantly evaluate the potential of a pruned sub-network.
Two Geometric Perspectives: Intra-Layer vs. Downstream Compensation
Researchers have developed two primary ways to redistribute the lost signal: compensating within the pruned layer itself, or compensating in the subsequent layer.
1. Intra-Layer Compensation and Neuron Merging
If two neurons produce highly correlated activations, removing one and reassigning its weights to the other can preserve the collective signal. Srinivas et al. (2015) pioneered this data-free approach by reassigning the outgoing weights of redundant neurons to similar remaining units.
Sra (2016) formalized this concept by using Determinantal Point Processes to select a diverse subset of neurons and reconstructing the activations of the pruned neurons via a least-squares projection onto the remaining ones.
Similarly, Kim et al. (2020) showed that under ReLU activation functions, representing pruned neurons as scaled versions of kept neurons allows the network to absorb the removed dimensions into the next layer's weights.
Naive Pruning:
[ Neuron A (Kept) ] ----> Output A
[ Neuron B (Pruned) ] --x (Signal Lost!)
Neuron Merging / Least-Squares Projection:
[ Neuron A (Kept) ] ----> Output A + Projection(Signal B)
By treating standard pruning as a special case of neuron merging, the network maintains the integrity of internal feature maps without requiring training data. This stabilization prevents the accuracy shocks common in standard pruning, allowing for deeper compression levels because the network does not need to relearn fundamental feature mappings from scratch (Shah et al. (2020)).
To scale this beyond one-to-one neuron replacement, Byrne et al. (2017) and Chau et al. (2023) proposed replacing entire groups of pruned neurons with a linear combination of a smaller "basis" set of neurons, using coefficients derived from orthogonal projection or linear programming to ensure the collective influence of the original layer is preserved.
2. Downstream Compensation
Alternatively, one can leave the pruned layer's weights alone and mathematically adjust the weights of the subsequent layer to absorb the missing inputs. Liang et al. (2021) utilized this two-step feature map reconstruction to ensure downstream feature maps remain as close as possible to the unpruned network.
More recently, Offergeld et al. (2024) showed that transforming neural network activations into an orthogonal subspace allows for the optimal recovery of lost information through linear least squares, enabling remaining weights to mimic the original output in a single step.
Data-Efficient and Training-Free Approximations
Because least-squares reconstruction relies on solving linear equations rather than iterative gradient descent, it is incredibly data-efficient. It does not require a full training dataset or labeled data; it only requires enough calibration samples to compute the activation covariance matrices.
For example, Chen et al. (2025) demonstrated that a closed-form analytical update can fully restore the performance of large autoregressive models using as few as 128 calibration samples. Similarly, Nova et al. (2023) showed that remaining weights can be mathematically scaled to minimize feature map loss using entirely unlabeled data, treating pruning as a geometric representative selection problem.
In scenarios where solving a full least-squares system is still too expensive, researchers use first-order statistics as a fast approximation:
- Bias Compensation: Zhao et al. (2024) showed that the average output of a removed channel can be integrated back into the model simply as a static bias term. This training-free compensation functions similarly to a rank-1 low-rank adaptation (LoRA), keeping the expected activation values of subsequent layers consistent.
- Mask Optimization: Instead of modifying the underlying weights, Kwon et al. (2022) kept the weights frozen and reconstructed the output signal by optimizing real-valued mask variables using a linear least-squares solver.
- Knowledge Recalling: Wu et al. (2025) leveraged low-rank transformations to reintegrate residual knowledge from discarded weights, preventing model collapse at extreme compression levels.
This localized, linear view of network adjustments extends beyond pruning. For instance, Sotoudeh et al. (2021) applied linear programming to network weight adjustments to "patch" specific errors with mathematical guarantees. By treating network repair as a localized linear optimization, they minimized weight changes using specific norms ($l_1$ or $l_\infty$), ensuring that modifications act as surgical repairs that do not degrade accuracy on previously learned data.
Ultimately, least-squares weight reconstruction works because the internal representations of deep networks are highly continuous and redundant. By framing compression as a localized signal-preservation task, we can replace hours of gradient-based fine-tuning with a single, exact linear algebra operation.
Key Foundational Work
An iterative pruning algorithm for feedforward neural networks (1997) — contributed 2 concept elements to this direction.
Data-free parameter pruning for Deep Neural Networks (2015) — contributed 1 concept elements to this direction.
Diversity Networks (2015) — contributed 1 concept elements to this direction.