Greedy Network Enlarging

Chuanjian Liu, Kai Han, An Xiao, Yiping Deng, Wei Zhang, Chunjing Xu, Yunhe WangarXiv 2021 · arXiv

Introduction

In deep learning, scaling up base architectures is a standard practice to improve performance. Popular methods like EfficientNet utilize a unified scaling rule, increasing depth, width, and resolution by a fixed ratio across all layers. However, this "one-size-fits-all" approach ignores the distinct computational requirements of different network stages. Early layers processing raw pixels have different needs than deeper layers extracting complex semantic features.

The Problem with Unified Scaling

Traditional scaling assumes that a balanced resource distribution in a small model should be maintained as it grows. Yet, empirical evidence suggests otherwise. For instance, in an EfficientNet-B4, later stages can consume significantly more MACs (Multiply-Accumulate operations) than earlier ones compared to the base B0 model. This imbalance suggests that a universal scaling ratio is sub-optimal and that we need a more fine-grained, "stage-wise" approach to resource allocation.

Greedy Network Enlarging (GNE)

To address this, the authors propose Greedy Network Enlarging (GNE). GNE moves away from global scaling rules in favor of a stage-wise, iterative enlargement process. The method is predicated on the "subcomponent assumption": an optimal large network likely contains an optimal smaller network as its core.

The GNE process: iteratively allocating MACs to increase width, depth, or resolution at specific stages.

The GNE algorithm follows a step-by-step approach:

  1. Iterative Growth: Starting from a base model, the network is grown in small increments of target MACs.
  2. Candidate Generation: At each step, multiple candidate architectures are generated by slightly increasing the width or depth of a specific stage, or by increasing the input resolution.
  3. Greedy Selection via Proxy Task: To avoid the prohibitive cost of full training, candidates are evaluated using a "fast performance evaluation." This proxy task involves training on a subset of ImageNet (100k images) for a limited number of epochs (20) with a pre-trained model to accelerate convergence.
  4. Progression: The best-performing candidate is selected as the new baseline for the next iteration.

Key Results and Efficiency

The models discovered through this process, prefixed with "S-" (e.g., S-EfficientNet, S-GhostNet), demonstrate superior efficiency over their traditionally scaled counterparts.

ImageNet Top-1 Accuracy vs. MACs: S-models consistently outperform standard EfficientNet and RegNet variants.

As shown in the results, S-EfficientNet-B1 achieves 79.9% Top-1 accuracy, surpassing the original B1 (79.1%) at the same computational cost. The gains are even more pronounced when applied to GhostNet. The S-GhostNet variant reached a state-of-the-art 84.3% accuracy at 4.4B MACs when combined with advanced training techniques like relabeling.

One interesting observation from the search process is that optimal input resolution does not increase linearly. Instead, it "rises wavily," indicating that the network's need for higher resolution fluctuates depending on its current width and depth. GNE allows the search to "fix" these imbalances dynamically by choosing the most effective dimension to scale at any given step.

Conclusion

Greedy Network Enlarging proves that where computational resources are allocated is just as critical as the total amount of computation. By allowing networks to grow organically stage-by-stage, GNE produces models that are both more accurate and more efficient than those following rigid scaling laws. While greedy algorithms can theoretically miss global optima, the empirical success of S-GhostNet and S-EfficientNet suggests the subcomponent assumption is a powerful heuristic for practical model design.

Summary updated

Abstract

Recent studies on deep convolutional neural networks present a simple paradigm of architecture design, i.e., models with more MACs typically achieve better accuracy, such as EfficientNet and RegNet. These works try to enlarge all the stages in the model with one unified rule by sampling and statistical methods. However, we observe that some network architectures have similar MACs and accuracies, but their allocations on computations for different stages are quite different. In this paper, we propose to enlarge the capacity of CNN models by improving their width, depth and resolution on stage level. Under the assumption that the top-performing smaller CNNs are a proper subcomponent of the top-performing larger CNNs, we propose an greedy network enlarging method based on the reallocation of computations. With step-by-step modifying the computations on different stages, the enlarged network will be equipped with optimal allocation and utilization of MACs. On EfficientNet, our method consistently outperforms the performance of the original scaling method. In particular, with application of our method on GhostNet, we achieve state-of-the-art 80.9% and 84.3% ImageNet top-1 accuracies under the setting of 600M and 4.4B MACs, respectively.
BibTeX
@misc{liu2021greedy,
  author = {Chuanjian Liu and Kai Han and An Xiao and Yiping Deng and Wei Zhang and Chunjing Xu and Yunhe Wang},
  title = {{Greedy Network Enlarging}},
  year = {2021},
  eprint = {2108.00177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.00177},
}
Figure