Deep neural networks are notoriously complex, non-linear systems. Their loss landscapes are riddled with local minima, and tracking how their predictions change during training usually requires computationally expensive simulation. Yet, a surprising property emerges when these networks become highly overparameterized: their training dynamics simplify dramatically. Under certain conditions, a deep network can be accurately approximated by a simple first-order Taylor expansion around its parameters.
This process, known as local linearization, transforms a black-box non-linear model into a tractable linear system. By treating the network's outputs as a linear function of its weights, researchers can bypass expensive retraining, compute analytical uncertainty bounds, and gain deep mechanistic insights into how deep learning models learn.
The Core Intuition: Why Non-Linear Networks Act Linearly
How can a function composed of hundreds of non-linear activation layers behave like a straight line? The answer lies in the geometry of overparameterized parameter spaces.
Consider a network $f(x; \theta)$ parameterized by weights $\theta$. If we perturb the weights by a small amount $\Delta \theta$, we can approximate the new output using a first-order Taylor series expansion around the initial weights $\theta_0$:
$$f(x; \theta_0 + \Delta \theta) \approx f(x; \theta_0) + \nabla_\theta f(x; \theta_0)^T \Delta \theta$$
In a standard, narrow network, this approximation breaks down quickly because the higher-order derivatives (the curvature of the landscape) are large, and the weights must move significantly to minimize the loss. However, as a network grows wider, a phenomenon known as "lazy training" occurs. In highly overparameterized regimes, the network requires only an infinitesimal shift in its weights $\Delta \theta$ to perfectly fit the training data. Because the parameter displacement is so small, the network stays within a local neighborhood where the loss landscape geometry is relatively simple and flat.
As demonstrated by Lee et al. in Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent, infinitely wide networks trained with gradient descent evolve predictably like Gaussian Processes. In this regime, the gradients of the network with respect to its weights at initialization serve as fixed, static features for the duration of training. This theoretical foundation is expanded upon by Misiakiewicz et al. in Six Lectures on Linearized Neural Networks, illustrating how overparameterized models effectively re-weight initial features rather than learning entirely new ones.
Gradients as Feature Maps
If a network behaves linearly with respect to its parameters, then the Jacobian matrix $\nabla_\theta f(x; \theta_0)$ effectively serves as a fixed feature map. This realization unlocks a powerful paradigm: instead of updating the deep layers of a network via backpropagation, we can extract the gradients of the model parameters and use them as static representations for linear models.
Standard Fine-Tuning:
Input (x) ──> [ Non-Linear Layers (Update Weights θ) ] ──> Prediction
Linearized Proxy:
Input (x) ──> [ Compute Fixed Jacobian ∇_θ f(x; θ₀) ] ──> [ Linear Classifier ] ──> Prediction
This approach bridges the gap between cheap linear probing and expensive full fine-tuning. For example, Mu et al. (Gradients as Features for Deep Representation Learning) combine activations and gradients into a specialized linear model, creating a local linear approximation of full fine-tuning that significantly outperforms standard linear probes across transfer learning tasks.
Similarly, Evci et al. (Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning) show that fine-tuning can be theoretically approximated as a linear combination of intermediate activations, proving that a linear classifier trained on a diverse set of frozen features can rival full parameter updates. For rapid adaptation, Maddox et al. (Fast Adaptation with Linearized Neural Networks) linearize pre-trained networks into "finite Neural Tangent Kernels," utilizing the model's Jacobian as features for a Gaussian Process to achieve stable, analytic adaptation without falling into local minima.
Predicting Training Dynamics and Performance
Because linearized models are governed by convex optimization, their behavior is mathematically predictable. This allows researchers to anticipate how a model will behave or perform without actually running the training loop.
For instance, Zancato et al. (Predicting Training Time Without Training) linearize pre-trained networks to estimate how many optimization steps a model needs to converge on a new task, achieving this 30–45x faster than actual training by solving low-dimensional stochastic differential equations.
This predictability also scales to massive architectures. Li et al. (Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach) discovered that large language models exhibit remarkably linear behavior in the loss landscape immediately following multitask training. By applying a first-order Taylor expansion, the authors can accurately predict an LLM's performance on various data subsets in seconds, bypassing the need for expensive repeated training.
Tractable Uncertainty and Bayesian Inference
Quantifying what a deep learning model doesn't know is notoriously difficult. True Bayesian Neural Networks (BNNs) require sampling from complex, high-dimensional weight posteriors, which is computationally prohibitive for modern architectures. Local linearization solves this by transforming the non-linear network into a Generalized Linear Model (GLM), where uncertainty can be propagated analytically.
As Immer et al. (Improving predictions of Bayesian neural nets via local linearization) demonstrate, common approximations like the Generalized Gauss-Newton (GGN) method implicitly treat the network as a GLM. By using this linearized model for predictions rather than the original non-linear one, they resolve chronic underfitting issues and dramatically improve uncertainty calibration.
This functional view is echoed by Rudner et al. (Tractable Function-Space Variational Inference in Bayesian Neural Networks), who utilize local linearization as a computationally efficient proxy for modeling functional posteriors directly in function space.
Furthermore, Tailor (Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence) demonstrates that linearizing a network with respect to its weights allows for the analytical identification of changepoints in residual ranks. This converts the computationally expensive task of full conformal prediction into a tractable geometric problem, calculating rigorous prediction intervals thousands of times faster than exact retraining methods.
Downstream Manipulation: Unlearning and Distillation
Beyond prediction and uncertainty, linearization makes complex model manipulation tasks mathematically tractable. Tasks like "machine unlearning" (forgetting specific training samples) or "dataset distillation" (compressing large datasets into a few synthetic samples) are typically hindered by the non-convexity of deep networks. Linearization restores convexity to these inner optimization loops.
In the context of machine unlearning, Block et al. (Machine Unlearning under Overparameterization) handle the non-linearity of modern networks by approximating the requirement to fit retained data using first-order Taylor expansions. This allows the unlearning problem to be solved efficiently as a constrained optimization task. To scale this to massive models, Li (Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models) shows that these linear approximations maintain high fidelity when restricted to a parameter subspace (such as batch normalization layers or visual prompts), enabling data removal at a fraction of the usual cost.
For dataset distillation, Loo et al. (Dataset Distillation with Convexified Implicit Gradients) linearize the network via a first-order Taylor expansion around its initialization. This approximation forces the inner optimization problem to become strongly convex, allowing researchers to compute implicit gradients reliably and distill datasets without getting trapped in chaotic, non-convex optimization paths.