TL;DR
This paper introduces the behavioral recovery rank—a geometric metric based on the behavioral Hessian—to quantify exactly how many parameter directions are needed to preserve a trained neural network's performance. The authors reveal a fundamental gap in model compression: standard magnitude and structural pruning fail to fully exploit a network's functional redundancy because this degeneracy is distributed across complex parameter directions rather than aligned with individual weights or neurons.
What Problem Does This Paper Address?
Modern neural networks are notoriously overparameterized, meaning they possess substantial functional redundancy—many weights can be altered or removed with little to no impact on the network's final output. While this redundancy forms the foundation for model compression (pruning), we still lack a clear understanding of the fundamental limits of compression.
Currently, achievable compression is usually measured post-hoc by applying a specific pruning algorithm. This paper shifts the focus to assessing redundancy as an intrinsic property of the model itself. The authors tackle two core questions:
- How many parameter directions are actually required to preserve a learned function?
- How much of this redundancy do standard pruning methods actually exploit?
Methods
To answer these questions, the authors propose a new framework:
- Behavioral Loss: Instead of comparing a model's predictions to ground-truth labels, the authors define a loss that compares a perturbed model's predictions to a reference trained model's predictions on a reference "probe set." This acts as a form of local self-distillation.
- Behavioral Hessian: Taking the second-order Taylor expansion of the behavioral loss yields the behavioral Hessian. Its leading eigenvectors identify the parameter directions that cause the largest changes in the model's behavior.
- Behavioral Recovery Rank: By projecting the trained model's weights onto the top-$k$ eigenspace of the behavioral Hessian, they find the minimum number of directions ($k$) needed to recover the model's original performance within a given tolerance. This serves as a geometric benchmark for ideal compression.
- Pruning Baselines: They compare this ideal benchmark against two standard one-shot, gradient-free pruning methods: magnitude pruning (removing the smallest individual weights) and structural pruning (greedily removing entire neurons).
Datasets
The methodology is evaluated in two distinct settings:
- Synthetic Nonlinear Teacher-Student Regression: A controlled setup using two-layer ReLU MLPs where a fixed teacher network ($10 \rightarrow 10 \rightarrow 10$) generates noiseless data, and student networks of varying hidden widths ($10 \rightarrow d \rightarrow 10$) are trained to match it. This provides a priori control over network capacity relative to task complexity.
- MNIST: The standard handwritten digit classification dataset, used to evaluate width-scaled MLPs ($784 \rightarrow w \rightarrow w \rightarrow 10$) on a real-world task where the true task complexity is unknown.
Key Insights and Ideas
- The Saturation Threshold: In the teacher-student setup, the behavioral recovery rank grows while the student is narrower than the teacher (capacity-limited regime) but flattens abruptly once the student width matches or exceeds the teacher width (task-saturated regime).
- Pruning's Blind Spot: Standard pruning methods completely miss this saturation point. The number of parameters they retain keeps growing linearly even after the task is fully saturated.
- Distributed Redundancy: Because individual behavioral eigendirections mix parameters across all hidden units, the gap between the ideal recovery rank and standard pruning implies that functional redundancy is distributed globally across parameter space. It is not exposed by individual weights or architectural coordinates, explaining why standard one-shot pruning is sub-optimal.
Key Results
- Teacher-Student Saturation: As shown in Figure 1, the behavioral recovery rank flattens at $d^* = 10$ (the teacher's width). In contrast, both magnitude and structural pruning continue to demand more degrees of freedom as network width increases.
- Sublinear Growth on MNIST: For MNIST (Figure 2), the behavioral recovery rank grows sublinearly with width. This indicates that while wider models do find additional behaviorally relevant directions, these directions expand much slower than the total parameter count.
- Loss-Optimal Ordering: Ordering the Hessian directions by their actual contribution to minimizing the loss (rather than just eigenvalue magnitude) halves the required recovery rank, revealing an even higher degree of latent degeneracy.
Limitations and Future Work
- Scalability: Computing repeated behavioral-Hessian eigensolves is computationally expensive, limiting the experiments to relatively small MLPs. Making this operational for large-scale models will require blockwise restrictions or probe-set subsampling.
- One-Shot Evaluation: The pruning baselines evaluated were "one-shot" and did not include retraining. The authors note that a full compression pipeline with retraining or weight refitting (which they explore briefly in Appendix F) can help close the gap.
- Predictive Theory: A promising future direction is using degeneracy measured on small models to theoretically bound the maximum achievable compression on much larger architectures once task complexity is saturated.
Summary updated