Introduction
In deep learning, scaling up base architectures is a standard practice to improve performance. Popular methods like EfficientNet utilize a unified scaling rule, increasing depth, width, and resolution by a fixed ratio across all layers. However, this "one-size-fits-all" approach ignores the distinct computational requirements of different network stages. Early layers processing raw pixels have different needs than deeper layers extracting complex semantic features.
The Problem with Unified Scaling
Traditional scaling assumes that a balanced resource distribution in a small model should be maintained as it grows. Yet, empirical evidence suggests otherwise. For instance, in an EfficientNet-B4, later stages can consume significantly more MACs (Multiply-Accumulate operations) than earlier ones compared to the base B0 model. This imbalance suggests that a universal scaling ratio is sub-optimal and that we need a more fine-grained, "stage-wise" approach to resource allocation.
Greedy Network Enlarging (GNE)
To address this, the authors propose Greedy Network Enlarging (GNE). GNE moves away from global scaling rules in favor of a stage-wise, iterative enlargement process. The method is predicated on the "subcomponent assumption": an optimal large network likely contains an optimal smaller network as its core.
The GNE algorithm follows a step-by-step approach:
- Iterative Growth: Starting from a base model, the network is grown in small increments of target MACs.
- Candidate Generation: At each step, multiple candidate architectures are generated by slightly increasing the width or depth of a specific stage, or by increasing the input resolution.
- Greedy Selection via Proxy Task: To avoid the prohibitive cost of full training, candidates are evaluated using a "fast performance evaluation." This proxy task involves training on a subset of ImageNet (100k images) for a limited number of epochs (20) with a pre-trained model to accelerate convergence.
- Progression: The best-performing candidate is selected as the new baseline for the next iteration.
Key Results and Efficiency
The models discovered through this process, prefixed with "S-" (e.g., S-EfficientNet, S-GhostNet), demonstrate superior efficiency over their traditionally scaled counterparts.
As shown in the results, S-EfficientNet-B1 achieves 79.9% Top-1 accuracy, surpassing the original B1 (79.1%) at the same computational cost. The gains are even more pronounced when applied to GhostNet. The S-GhostNet variant reached a state-of-the-art 84.3% accuracy at 4.4B MACs when combined with advanced training techniques like relabeling.
One interesting observation from the search process is that optimal input resolution does not increase linearly. Instead, it "rises wavily," indicating that the network's need for higher resolution fluctuates depending on its current width and depth. GNE allows the search to "fix" these imbalances dynamically by choosing the most effective dimension to scale at any given step.
Conclusion
Greedy Network Enlarging proves that where computational resources are allocated is just as critical as the total amount of computation. By allowing networks to grow organically stage-by-stage, GNE produces models that are both more accurate and more efficient than those following rigid scaling laws. While greedy algorithms can theoretically miss global optima, the empirical success of S-GhostNet and S-EfficientNet suggests the subcomponent assumption is a powerful heuristic for practical model design.
Summary updated