TL;DR: GradMax is a novel method for "growing" neural networks during training by adding new neurons initialized to maximize their gradient norms. By using Singular Value Decomposition (SVD) to find the optimal initialization in closed form, GradMax improves training dynamics and final model performance while avoiding the massive computational costs of training large architectures from scratch.
Introduction: The Challenge of Network Growth
In standard deep learning, practitioners typically select a fixed architecture (like ResNet-50) and train its parameters from scratch. If a different capacity is needed, the entire training process must be restarted, which is highly inefficient. An alternative paradigm is "growing" networks—starting with a small model and dynamically adding neurons as needed.
However, the mechanics of growing a network are tricky. If new neurons are added randomly, they might not contribute meaningfully to the learning process initially. Conversely, a "greedy" approach that adds neurons to immediately reduce the training loss often yields only a short-term boost, failing to accelerate long-term training. The authors identify a critical gap: we need an initialization strategy for new neurons that preserves what the network has already learned while "supercharging" the subsequent training dynamics.
The GradMax Method
To address this, the authors introduce GradMax (Gradient Maximizing Growth). The core idea is to add $k$ new neurons to a layer such that the norm of the gradients for these new weights is maximized, rather than focusing on immediate loss reduction.
The GradMax initialization process follows these key steps:
- Preserve Output: When adding new neurons, their incoming weights are initialized to zero. This ensures that the network's output remains exactly the same at the moment of growth, perfectly preserving previously learned information.
- Maximize Gradients: The authors formulate an optimization problem to find the outgoing weights that result in the largest possible gradient norm for the new units.
- SVD Solution: Remarkably, under common assumptions (such as using ReLU or Tanh activations), this maximization problem has a closed-form solution. By performing a Singular Value Decomposition (SVD) on the gradient of the loss with respect to the layer's activations, the top $k$ left-singular vectors can be used to optimally initialize the new weights.
- Efficiency: Instead of calculating the SVD over the entire dataset, a large minibatch provides a sufficiently accurate and computationally efficient approximation.
Key Insights and Results
The researchers evaluated GradMax across several scales, from synthetic Teacher-Student tasks using MLPs to large-scale benchmarks like ImageNet using MobileNet-v1. GradMax consistently outperformed random initialization and existing growth methods like "Firefly."
- Gradients over Loss: The most surprising insight is that maximizing the potential for change (gradient norm) is more effective for long-term training than immediately reducing the loss. This "non-greedy" approach leads to better final accuracy.
- Persistence of Norms: The high gradient norm injected by GradMax isn't just a momentary spike; it persists for many training steps, leading to faster overall convergence.
- ImageNet Performance: When growing a MobileNet-v1 from 25% width to 100% width, GradMax achieved 68.6% top-1 accuracy, outperforming Random growth (66.9%) and Firefly (66.4%).
- Computational Efficiency: Training via GradMax used only about 75% of the FLOPs required to train the full baseline model from scratch, while significantly narrowing the accuracy gap compared to other growth methods. On CIFAR-10 with Wide-ResNet, GradMax reached 91.1% accuracy, coming closer to the trained-from-scratch baseline (92.9%) than any other strategy.
Limitations and Future Work
While highly effective, GradMax has a few limitations. The current mathematical derivation requires activation functions that pass through the origin ($f(0)=0$) and have a non-zero derivative there (like ReLU and Tanh). Adaptations may be needed for other activation types. Furthermore, while GradMax dictates how to initialize, the optimal schedule (when to grow) and location (which layer to grow) remain fixed in this study. The authors suggest that singular values could eventually be used to automate these architectural decisions. Finally, extending this logic beyond MLPs and CNNs to Transformers or attention-based mechanisms remains an exciting open area for research.
Summary updated