Abstract
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights and find the optimal initialization efficiently by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.
1. Introduction
In contemporary deep learning, model architectures are typically static throughout the training process. If a chosen configuration—defined by the number of layers and neurons—proves suboptimal, practitioners must often restart training from scratch with a revised architecture, a process that is both time-consuming and computationally expensive.
"Growing" neural networks by adding capacity as needed presents a compelling alternative. However, the primary challenge lies in the initialization of new neurons. Conventional approaches, such as random initialization or greedy loss-reduction strategies, often fail to optimize for long-term training performance. This paper introduces GradMax, a method that leverages gradient information to initialize new neurons in a way that maximizes their potential for future learning.
2. Methodology: Gradient Maximizing Growth
The GradMax algorithm is built upon two core principles: function preservation and gradient maximization.
2.1 Function Preservation
To ensure that adding a neuron does not disrupt the features already learned by the network, GradMax maintains the network's output at the moment of growth. This is achieved by initializing the incoming weights of the new neurons ($W_\ell^{\text{new}}$) to zero. Consequently, the new neurons initially contribute nothing to the output, preserving the model's current performance.
2.2 Gradient Maximization
While the immediate output is unchanged, GradMax seeks to maximize the potential for the new neurons to learn quickly in subsequent iterations. This is done by selecting outgoing weights ($W_{\ell+1}^{\text{new}}$) that maximize the norm of the gradient of the loss with respect to the new weights.
The authors derive a closed-form solution for this maximization problem using Singular Value Decomposition (SVD). By analyzing the gradient of the loss with respect to an auxiliary weight matrix, the method identifies the directions of steepest descent in the weight space. The new neurons are then initialized using the top $k$ singular vectors, ensuring they are positioned to capture the most significant remaining signals in the data.
3. Experimental Results
The effectiveness of GradMax was validated across various architectures and datasets, including synthetic Teacher-Student MLPs, CIFAR-10/100 (using VGG-11 and Wide-ResNet), and ImageNet (using MobileNet-v1).
3.1 Performance Benchmarks
GradMax consistently outperformed baseline growth methods. On ImageNet with MobileNet-v1, GradMax achieved a test accuracy of 68.6%, compared to 66.9% for random growth and 66.4% for the Firefly method. In CIFAR-10 experiments with Wide-ResNet, GradMax (91.1% accuracy) significantly outperformed other growing strategies, approaching the performance of a "Baseline-Big" model (92.9%) that was trained at full capacity from the start.
3.2 Computational Efficiency
A key advantage of GradMax is its efficiency. By following a linear growth schedule, the method can achieve performance levels near those of large static models while utilizing only approximately 75% of the total floating-point operations (FLOPs). Furthermore, the high-gradient state induced by GradMax persists for many iterations, leading to faster overall convergence compared to greedy or random alternatives.
4. Conclusion and Future Work
GradMax demonstrates that maximizing the initial gradient norm of new neurons is a superior strategy for growing neural networks compared to immediate loss reduction. While currently limited to activation functions that map zero to zero (e.g., ReLU, Tanh), future research could extend this framework to a broader class of activations and more complex architectures like Transformers. Additionally, the singular values obtained during the SVD process could serve as a dynamic signal to determine the optimal timing and location for network growth.
Summary updated