Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation

Adrian Goldwaser, Hong GeSubmission to ICLR 2024 · Rejected

TL;DR

Neural Tangent Kernel (NTK) theory assumes deep networks operate in a "lazy" regime where internal representations remain fixed during training, whereas real networks learn rich features. This paper introduces iterative linearisation, a framework that freezes feature representations for $K$ optimization steps at a time to control how frequently features are updated. Surprisingly, the authors find that updating features as few as 8 to 12 times throughout the entire training run is sufficient to match standard SGD generalization performance.

What Problem Does This Paper Address?

While infinitely wide neural networks can be tractably analyzed using the Neural Tangent Kernel (NTK), their empirical kernel remains static during gradient flow—meaning they do not learn intermediate features. In contrast, practical finite-width networks undergo significant representation learning, and their empirical NTK changes substantially during training.

This leaves fundamental questions open: How much feature learning is actually necessary for a network to generalize well? Does standard gradient descent update features far more often than necessary?

Methods

The authors introduce iterative linearisation, which decouples parameter updates from feature updates:

  • Proxy Model: At step $s$, the neural network is approximated by its first-order Taylor expansion around parameters $\theta_s$: $$f_{s,t}^{\text{lin}}(x) = f_{\theta_s}(x) + \nabla_\theta f_{\theta_s}(x)^\top (\theta_t - \theta_s)$$
  • Periodic Re-linearisation: Parameters $\theta$ are trained on this proxy linear model for $K$ gradient steps while keeping the Jacobian (the features $\phi = \nabla_\theta f_{\theta_s}$) fixed. Every $K$ steps, the model is re-linearised at the new weights $\theta_{s+K}$.
  • $K = 1$ recovers standard gradient descent (continuous feature learning).
  • $K = \infty$ recovers purely linearised training (fixed random features / lazy regime).
  • Intermediate values of $K$ interpolate between the rich and lazy regimes.

Connections to Second-Order Methods

The authors show that if the linearised model is trained to convergence at each re-linearisation step under mean squared error, iterative linearisation becomes mathematically identical to the Gauss-Newton algorithm. Furthermore, adding damping $\lambda$ to Gauss-Newton acts analogously to choosing a smaller step period $K$: it restricts how far parameters travel on a single linearised surface, effectively forcing more frequent feature updates.

Datasets

  • CIFAR-10: Evaluated on image classification using a modified LeNet architecture (two convolutional layers followed by dense layers) and ResNet-18 (with and without Batch Normalization and data augmentation).
  • Synthetic 1D Quintic Polynomial: $y = \frac{1}{32}x^5 - \frac{1}{2}x^3 + 2x - 1$ bounded on $[-4, 4]$ (20 uniform training points, 1,000 test points) evaluated on a 5-layer MLP to test damping behavior in Gauss-Newton optimization.

Key Insights and Ideas

  • Features need remarkably few updates: Standard SGD updates features every single mini-batch (hundreds of thousands of times). In contrast, refreshing features just 8–12 times across the entire training trajectory achieves test accuracy nearly identical to standard SGD.
  • Feature learning is still mandatory: Setting $K = \infty$ (pure NTK / frozen features) yields much worse generalization and scales poorly as dataset size increases, showing that some feature updates are strictly necessary.
  • Premature convergence hurts generalization: When Adam was used to rapidly solve each linearised proxy problem to 100% train accuracy before re-linearising, test accuracy plummeted from 69.4% to 61.3%. Training loss converged faster than the network could update its representations.
  • A feature-learning view of damping: Damping in second-order optimization is usually justified by numerical stability. Here, the authors demonstrate that damping prevents the optimizer from overcommitting to outdated features, directly improving generalization by encouraging smoother function fits.

Key Results

  • SGD-level performance with sparse updates: On CIFAR-10 with data augmentation, standard SGD ($K=1$) reached 82.3% test accuracy, while iterative linearisation with $K = 5 \times 10^6$ (only ~8–12 re-linearisations) reached 81.4%—a gap of under 1%.
  • Architecture generality: Similar parity was demonstrated on full-batch GD, mini-batch SGD, and ResNet-18 architectures (with and without BatchNorm).
  • Linear Probing: Evaluating intermediate representations with linear probes revealed that models with large $K$ accumulate feature representations more slowly and reach lower probe accuracy than standard SGD, yet this smaller amount of feature learning is still sufficient for final task accuracy.

Limitations and Future Work

  • Compute and Scale: Because large-$K$ linearisation requires very small learning rates to maintain stability, experiments demanded substantial compute (~1,500 GPU hours) and were restricted to CIFAR-10, small CNNs, and ResNet-18. It remains an open question whether these findings translate to large Vision Transformers or Large Language Models (LLMs).
  • Fixed vs. Adaptive Periodicity: The paper only studies fixed intervals of $K$. Since empirical NTK kernels are known to evolve fastest in the earliest phase of training, future work could explore adaptive schedules (e.g., small $K$ early, large $K$ late) to develop more computationally efficient training and fine-tuning algorithms.

Summary updated

Abstract

Larger and deeper networks generalise well despite their increased capacity to overfit. Understanding why this happens is theoretically and practically important. One recent approach looks at the infinitely wide limits of such networks and their corresponding kernels. However, these theoretical tools cannot fully explain finite networks as the empirical kernel changes significantly during gradient-descent-based training in contrast to infinite networks. In this work, we derive an iterative linearised training method as a novel empirical tool to further investigate this distinction, allowing us to control for sparse (i.e. infrequent) feature updates and quantify the frequency of feature learning needed to achieve comparable performance. We justify iterative linearisation as an interpolation between a finite analog of the infinite width regime, which does not learn features, and standard gradient descent training, which does. Informally, we also show that it is analogous to a damped version of the Gauss-Newton algorithm --- a second-order method. We show that in a variety of cases, iterative linearised training surprisingly performs on par with standard training, noting in particular how much less frequent feature learning is required to achieve comparable performance. We also show that feature learning is essential for good performance. Since such feature learning inevitably causes changes in the NTK kernel, we provide direct negative evidence for the NTK theory, which states the NTK kernel remains constant during training.
BibTeX
@inproceedings{goldwaser2022understanding,
  author = {Adrian Goldwaser and Hong Ge},
  title = {{Understanding Sparse Feature Updates in Deep Networks using Iterative Linearisation}},
  booktitle = {OPT @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=wpxn_MdPg-M},
}
Figure