When optimizing, pruning, or modifying a deep neural network, we often need to know how sensitive the model's loss is to changes in its weights. This sensitivity is dictated by the local geometry of the loss landscape, captured mathematically by second-order derivatives: the Hessian matrix (which measures loss curvature) or its close relative, the Fisher Information Matrix (which measures how parameter changes affect the output distribution).
For a network with $N$ parameters, these matrices are massive ($N \times N$). Storing or inverting them requires $O(N^2)$ or $O(N^3)$ resources, which is completely impractical for modern models. To bypass this computational bottleneck, the standard engineering workaround is simple: ignore everything except the diagonal elements.
This diagonal approximation assumes that every weight in a neural network acts independently. While computationally convenient, a growing body of research demonstrates that this independence assumption is fundamentally flawed, leading to severe performance degradation across a wide range of tasks.
The Geometry of Parameter Independence
To understand why the diagonal approximation fails, imagine a two-dimensional loss landscape shaped like a narrow, tilted valley.
Diagonal Approximation True Loss Landscape
(Axes-aligned Ellipse) (Tilted Ellipse)
Parameter 2 Parameter 2
▲ ▲
│ ┌─┐ │ ▲
│ ┌─┘ └─┐ │ ╱ ╲
│ │ │ │ ╱ ╲
│ └─┐ ┌─┘ │ ╲ ╱
│ └─┘ │ ╲ ╱
└──────────► Parameter 1 └────▼─────► Parameter 1
If you only look at the diagonal elements of the Hessian, you are measuring the curvature strictly along the coordinate axes (Parameter 1 and Parameter 2 independently). Along both axes, the slope changes moderately, so a diagonal approximation concludes that both parameters are moderately sensitive.
However, the true loss landscape is a tilted ellipse. The sharpest ascent and the flattest descent do not align with the individual parameter axes; they are combinations of both parameters.
- The off-diagonal terms capture these cross-parameter correlations.
- Neglecting them means you cannot see that moving both parameters together allows you to slide smoothly along the bottom of the valley, while moving them in opposition pushes you up a steep cliff.
In high-dimensional neural networks, parameters are highly collaborative. Features are distributed across layers, channels, and attention heads. Assuming they do not interact creates a distorted map of the loss landscape.
Where the Independence Assumption Fails
The neglect of parameter correlations causes systematic failures in several core areas of deep learning.
1. Continual Learning and Catastrophic Forgetting
In continual learning, a model learns multiple tasks sequentially. To prevent the model from forgetting Task 1 while learning Task 2, algorithms like Elastic Weight Consolidation (EWC), introduced by Kirkpatrick et al., calculate parameter importance using a diagonal Fisher matrix and penalize changes to "important" weights.
However, Kirkpatrick et al. note that this diagonal approximation underestimates the true uncertainty of the weights. Because it ignores off-diagonal correlations, the model gradually overwrites critical shared pathways, leading to catastrophic forgetting over extended task sequences. As Zekun Wang et al. demonstrate in diffusion models, the curvature of the loss landscape is often concentrated almost entirely in the off-diagonal terms. Relying solely on diagonal importance measures can result in a near-total misestimation of parameter sensitivity.
2. Network Pruning and Quantization
Compressing networks requires identifying which parameters are redundant. Classic pruning frameworks like Optimal Brain Damage by LeCun et al. and Early Brain Damage by Tresp et al. rely on diagonal Hessian approximations to rank weight importance.
The core limitation, as argued by LeCun et al., is that neglecting the cross-terms of the second derivative leads to inaccurate importance rankings when parameters are highly interdependent. A weight might appear unimportant on its own (low diagonal value), but it may be performing vital error-correction for a neighboring weight (high off-diagonal correlation). Deleting it breaks the partnership, degrading model accuracy. Similarly, in post-training quantization, Choi et al. and Kim et al. show that ignoring these intricate inter-parameter and channel-wise dependencies causes a significant loss of precision when rounding weights to lower bit-widths.
3. Machine Unlearning and Data Erasure
With regulations like GDPR's "right to be forgotten," researchers use second-order updates to efficiently "erase" the influence of specific training samples without retraining from scratch.
Peste et al. demonstrate that the inclusion of off-diagonal terms in the Fisher Information Matrix is absolutely critical for effective data erasure. When these non-diagonal components are ignored, the unlearning algorithm fails to account for how parameters jointly encoded the targeted data. This results in either incomplete erasure (leaving residual traces of the data) or a severe degradation of the model's remaining performance on unrelated data.
4. Optimization and Uncertainty Estimation
Popular optimizers like Adam track the running average of squared gradients to scale learning rates. As Dangel et al. point out, this effectively recycles optimizer states to approximate the diagonal of the Fisher matrix for free.
However, Dahlinger et al. argue that restricting curvature estimation to the diagonal prevents optimizers from fully exploiting the true geometric structure of the loss surface, leading to suboptimal step directions in sharp or highly tilted regions. Furthermore, for tasks like out-of-distribution detection and Bayesian inference, Bergamin et al. and Krämer et al. show that ignoring off-diagonal correlations severely degrades model calibration and anomaly detection sensitivity, as the model cannot accurately quantify its own predictive uncertainty.
Moving Beyond the Diagonal
To fix these limitations without incurring prohibitive computational costs, researchers have developed several alternative structural approximations that capture off-diagonal interactions efficiently.
- Coordinate Rotations: Instead of changing the mathematical operator, why not change the space? Liu et al. introduce a method that applies a clever reparameterization (a rotation) to the network's layers. This rotation aligns the parameter space so that the true Fisher Information Matrix becomes approximately diagonal, allowing standard diagonal algorithms to capture the essential curvature information accurately.
- Kronecker Factorization: Rather than treating every parameter independently, Kronecker-factored approximations (like EKFAC by George et al.) treat weight matrices as blocks. They model the correlations between inputs and outputs within a layer, capturing significant off-diagonal structure while maintaining scalability.
- Low-Rank Approximations: In certain regimes, the gradients of overparameterized models become highly collinear. Wang et al. exploit this property in diffusion models to construct a rank-1 Fisher approximation. This captures the dominant direction of cross-parameter curvature without the memory overhead of a full matrix.
By moving beyond the strict independence assumption, these approaches allow models to preserve knowledge, compress cleanly, and navigate complex loss landscapes exactly as they are shaped: as deeply interconnected webs of parameters.
Key Foundational Work
Optimal Brain Damage (1989) — contributed 1 concept elements to this direction.
Early Brain Damage (1996) — contributed 1 concept elements to this direction.