Network Expansion For Practical Training Acceleration

Ning Ding, Yehui Tang, Kai Han, Chao Xu, Yunhe Wang

Abstract

Training massive neural networks is notoriously slow and expensive. This paper introduces Network Expansion, a "grow-as-you-go" strategy that starts training with a small, sparse sub-network and gradually expands it into a full-sized model. By using clever initialization techniques like filter orthogonality for CNNs and EMA-based layer insertion for Vision Transformers (ViTs), the authors achieve up to a 1.42× practical speedup on standard GPUs with almost no loss in accuracy.

Introduction

As deep learning models (like ResNets and Vision Transformers) and datasets (like ImageNet) grow in scale, the time and financial cost of training them from scratch have become prohibitive. While many researchers focus on making models efficient for inference, making the training process faster is just as critical.

Existing "efficient training" methods often fail to reduce "wall-clock" time on standard GPUs or are designed specifically for low-power edge devices. The authors propose Network Expansion to provide real-world acceleration on standard hardware without requiring specialized libraries or extensive hyperparameter tuning.

Methodology

The core idea is to start small and grow. The training process is divided into stages: starting with a "slim" version of the target architecture and gradually "expanding" the model by adding parameters until it reaches its full, dense size.

Overview of the multi-stage width expansion process, where a model starts with fewer channels and grows over time.

Simply adding random parameters often leads to redundancy. To solve this, two specific strategies were developed:

Width Expansion for CNNs

In CNNs, more channels (filters) are gradually added to each layer. To ensure new filters learn useful, non-redundant features, the authors employ Filter Orthogonality. Using Singular Value Decomposition (SVD), new weights are initialized to be mathematically orthogonal to existing ones, forcing the new channels to capture diverse information.

Depth Expansion for ViTs

For Vision Transformers, more transformer blocks (layers) are added. Instead of just copying existing layers, which introduces redundancy, the authors maintain an Exponential Moving Average (EMA) of the model weights.

Comparison of depth expansion strategies: (a) StackBERT's copy-and-stack approach vs. (b) the proposed EMA-based layer insertion method.

As illustrated in the figure above, layers from the EMA version are "inserted" into the active model during expansion. This provides a "parameter bank" of well-learned features that are slightly different from the current weights, helping the deeper model converge faster.

Experimental Results

The approach was validated on industry-standard benchmarks including CIFAR-10/100, ImageNet-1k, and Cityscapes. Key findings include:

  • ResNet-101 on ImageNet: Achieved a 1.42× speedup in total training time with only a 0.2% drop in Top-1 accuracy.
  • DeiT-base (ViT) on ImageNet: Achieved a 1.34× speedup, saving roughly 120 GPU-hours for a single training run.
  • Downstream Tasks: A ResNet-101 backbone trained via expansion and transferred to semantic segmentation (DeepLabV3+) performed within 0.5% mIoU of the baseline while being 1.3× faster to train.

Key Insights

  • Sparsity as a Training Tool: Leveraging sparsity during training is a viable path to practical acceleration.
  • Initialization is Crucial: The relationship between old and new parameters matters. Orthogonality in CNNs and EMA-based "soft" copying in ViTs are keys to maintaining model expressiveness during growth.
  • Practicality: The method integrates easily into standard frameworks like PyTorch and TensorFlow without needing custom CUDA kernels.

Limitations

There is a floor to how small the initial sub-network can be; starting with only 12.5% of channels can lead to significant accuracy degradation. Furthermore, finding the optimal frequency for expansion remains somewhat empirical, particularly for Vision Transformers.

Summary updated

Abstract

Recently, the sizes of deep neural networks and training datasets both increase drastically to pursue better performance in a practical sense. With the prevalence of transformer-based models in vision tasks, even more pressure is laid on the GPU platforms to train these heavy models, which consumes a large amount of time and computing resources as well. Therefore, it's crucial to accelerate the training process of deep neural networks. In this paper, we propose a general network expansion method to reduce the practical time cost of the model training process. Specifically, we utilize both width- and depth-level sparsity of dense models to accelerate the training of deep neural networks. Firstly, we pick a sparse sub-network from the original dense model by reducing the number of parameters as the starting point of training. Then the sparse architecture will gradually expand during the training procedure and finally grow into a dense one. We design different expanding strategies to grow CNNs and ViTs respectively, due to the great heterogeneity in between the two architectures. Our method can be easily integrated into popular deep learning frameworks, which saves considerable training time and hardware resources. Extensive experiments show that our acceleration method can significantly speed up the training process of modern vision models on general GPU devices with negligible performance drop (e.g. 1.42× faster for ResNet-101 and 1.34× faster for DeiT-base on ImageNet-1k). The code is available at https://github.com/huawei-noah/Efficient-Computing/tree/master/TrainingAcceleration/NetworkExpansion and https://gitee.com/mindspore/hub/blob/master/mshub_res/assets/noah-cvlab/gpu/1.8/networkexpansion_v1.0_imagenet2012.md
BibTeX
@inproceedings{ding2023network,
  author = {Ning Ding and Yehui Tang and Kai Han and Chao Xu and Yunhe Wang},
  title = {{Network Expansion For Practical Training Acceleration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01941},
}
Figure