Abstract
Neural growth is the process of growing a small neural network to a large network and has been utilized to accelerate the training of deep neural networks. One crucial aspect of neural growth is determining the optimal growth timing. However, few studies investigate this systematically. Our study reveals that neural growth inherently exhibits a regularization effect, whose intensity is influenced by the chosen policy for growth timing. While this regularization effect may mitigate the overfitting risk of the model, it may lead to a notable accuracy drop when the model underfits. Yet, current approaches have not addressed this issue due to their lack of consideration of the regularization effect from neural growth. Motivated by these findings, we propose an under/over fitting risk-aware growth timing policy, which automatically adjusts the growth timing informed by the level of potential under/overfitting risks to address both risks. Comprehensive experiments conducted using CIFAR-10/100 and ImageNet datasets show that the proposed policy achieves accuracy improvements of up to 1.3% in models prone to underfitting while achieving similar accuracies in models suffering from overfitting compared to the existing methods.
Introduction: The Challenge of Growth Timing
Training massive Deep Neural Networks (DNNs) is computationally expensive. "Neural growth" has emerged as a solution: starting training with a shallow "seed" network and gradually adding layers until the target depth is reached. This approach saves significant computational resources because the initial training phases occur on a much smaller, faster model.
However, a critical question remains: When exactly should the next layer be added? Existing strategies typically follow fixed periodic schedules or wait for the current model to converge. This paper identifies a major oversight in these methods: they ignore the fact that the growth process itself acts as a form of regularization. Because layers added later are exposed to fewer training epochs, they are less likely to "memorize" noise. While beneficial for preventing overfitting on small datasets, this regularization can lead to significant accuracy drops (underfitting) on complex datasets like ImageNet.
Methods: The FRAGrow Policy
To address these challenges, the authors propose FRAGrow (Fitting Risk-Aware Grow policy). This policy monitors the model's "fitting risk" in real-time and dynamically adjusts the growth interval.
Overfitting Risk Level (ORL)
The core metric used is the Overfitting Risk Level, defined as: $$ORL = \text{Training Accuracy} - \text{Validation Accuracy}$$ A high ORL indicates the model is beginning to overfit, while a low ORL suggests it is struggling to capture the underlying data patterns (underfitting).
Dynamic Growth Adjustment
Instead of a static interval, FRAGrow calculates the number of epochs to wait before adding a new layer using a sigmoid-based function. This allows the system to self-correct:
- Underfitting (Low ORL): The growth interval shrinks. The model grows faster, reducing the regularization effect and allowing the model to gain capacity quickly.
- Overfitting (High ORL): The growth interval expands. The model grows slower, increasing the regularization effect to improve generalization.
Key Insights: Growth as Regularization
The fundamental insight of this work is that neural growth inherently limits the "exposure time" of new layers to the training data. If a model grows too slowly on a difficult task, it becomes over-regularized. Conversely, if it grows too quickly on a simple task, it may overfit. By treating growth timing as a primary lever for controlling generalization error, FRAGrow provides a closed-loop system that "reads the room" and adjusts its schedule without prior knowledge of the dataset's complexity.
Experimental Results
The researchers evaluated FRAGrow across CIFAR-10/100 and ImageNet benchmarks using various architectures (VGG, ResNet, MobileNetV2).
- Accuracy Improvements: On ImageNet, where underfitting is common, FRAGrow outperformed existing growth baselines by 0.47% for ResNet and 1.31% for VGG.
- Training Efficiency: Compared to "vanilla" training, FRAGrow reduced training time by approximately 20% on CIFAR and 10% on ImageNet while maintaining or improving final accuracy.
- Robustness: The policy proved effective across single-branch, multiple-branch, and lightweight architectures, demonstrating its broad applicability.
Conclusion
FRAGrow demonstrates that growth timing is not just a scheduling detail but a powerful regularization mechanism. By making growth "risk-aware," researchers can achieve the efficiency benefits of neural growth without sacrificing accuracy on complex tasks. Future work will explore extending this policy to dense prediction tasks like object detection and semantic segmentation.
Summary updated