Abstract
Training massive neural networks is notoriously slow and expensive. This paper introduces Network Expansion, a "grow-as-you-go" strategy that starts training with a small, sparse sub-network and gradually expands it into a full-sized model. By using clever initialization techniques like filter orthogonality for CNNs and EMA-based layer insertion for Vision Transformers (ViTs), the authors achieve up to a 1.42× practical speedup on standard GPUs with almost no loss in accuracy.
Introduction
As deep learning models (like ResNets and Vision Transformers) and datasets (like ImageNet) grow in scale, the time and financial cost of training them from scratch have become prohibitive. While many researchers focus on making models efficient for inference, making the training process faster is just as critical.
Existing "efficient training" methods often fail to reduce "wall-clock" time on standard GPUs or are designed specifically for low-power edge devices. The authors propose Network Expansion to provide real-world acceleration on standard hardware without requiring specialized libraries or extensive hyperparameter tuning.
Methodology
The core idea is to start small and grow. The training process is divided into stages: starting with a "slim" version of the target architecture and gradually "expanding" the model by adding parameters until it reaches its full, dense size.
Simply adding random parameters often leads to redundancy. To solve this, two specific strategies were developed:
Width Expansion for CNNs
In CNNs, more channels (filters) are gradually added to each layer. To ensure new filters learn useful, non-redundant features, the authors employ Filter Orthogonality. Using Singular Value Decomposition (SVD), new weights are initialized to be mathematically orthogonal to existing ones, forcing the new channels to capture diverse information.
Depth Expansion for ViTs
For Vision Transformers, more transformer blocks (layers) are added. Instead of just copying existing layers, which introduces redundancy, the authors maintain an Exponential Moving Average (EMA) of the model weights.
As illustrated in the figure above, layers from the EMA version are "inserted" into the active model during expansion. This provides a "parameter bank" of well-learned features that are slightly different from the current weights, helping the deeper model converge faster.
Experimental Results
The approach was validated on industry-standard benchmarks including CIFAR-10/100, ImageNet-1k, and Cityscapes. Key findings include:
- ResNet-101 on ImageNet: Achieved a 1.42× speedup in total training time with only a 0.2% drop in Top-1 accuracy.
- DeiT-base (ViT) on ImageNet: Achieved a 1.34× speedup, saving roughly 120 GPU-hours for a single training run.
- Downstream Tasks: A ResNet-101 backbone trained via expansion and transferred to semantic segmentation (DeepLabV3+) performed within 0.5% mIoU of the baseline while being 1.3× faster to train.
Key Insights
- Sparsity as a Training Tool: Leveraging sparsity during training is a viable path to practical acceleration.
- Initialization is Crucial: The relationship between old and new parameters matters. Orthogonality in CNNs and EMA-based "soft" copying in ViTs are keys to maintaining model expressiveness during growth.
- Practicality: The method integrates easily into standard frameworks like PyTorch and TensorFlow without needing custom CUDA kernels.
Limitations
There is a floor to how small the initial sub-network can be; starting with only 12.5% of channels can lead to significant accuracy degradation. Furthermore, finding the optimal frequency for expansion remains somewhat empirical, particularly for Vision Transformers.
Summary updated