ScaleNet: Searching for the Model to Scale

Jiyang Xie, Xiu Su, Shan You, Zhanyu Ma, Fei Wang, Chen Qian

Introduction

In the modern deep learning landscape, the demand for neural networks spans a vast spectrum of hardware capabilities. A single architecture rarely suffices; instead, researchers require a "family" of models tailored for diverse environments, ranging from resource-constrained wearables to high-performance server clusters. Historically, achieving this variety has relied on two distinct approaches: one-shot Neural Architecture Search (NAS) or manual scaling heuristics like those used in EfficientNet.

However, both methods possess inherent limitations. One-shot NAS typically optimizes for a specific computational budget, failing to provide a clear path for scaling without re-initiating an expensive search. Conversely, manual scaling often applies rigid rules (adjusting depth, width, and resolution) to a fixed base model, ignoring the possibility that the optimal architecture for a small budget may not be the ideal foundation for a larger one. ScaleNet addresses these challenges by introducing a framework that jointly searches for an optimal base architecture and its unique, architecture-dependent scaling strategy.

Methodology

ScaleNet bridges the gap between architecture search and model scaling through four primary innovations designed to automate the discovery of scalable neural networks.

1. The Super-supernet

While standard NAS utilizes a "supernet" to represent a range of sub-networks, ScaleNet employs a "super-supernet." This structure is designed to embody models across a massive spectrum of FLOPs, providing a unified search space that encompasses both the architectural parameters and the scaling dimensions.

2. Hierarchical Sampling Strategy (HSS)

Training a supernet that covers a wide range of sizes is computationally difficult. Standard random sampling often biases the training toward "middle-sized" models, leaving the extremes (very small and very large configurations) undertrained. To ensure fairness and accuracy across all potential scales, ScaleNet implements HSS. This strategy uses a multi-modal distribution to ensure that models at every scaling stage (from S0 to S3) receive effective and balanced training.

3. Markov Chain-based Evolution Algorithm (MCEA)

To navigate the complex search space, the authors propose MCEA. This algorithm models the search as a coupled Markov chain where the base architecture and the scaling strategy (the specific ratios for increasing depth, width, and resolution) evolve in tandem. This ensures that the resulting scaling laws are tailor-made for the specific structural strengths of the discovered base model.

4. Generalization and Extrapolation

ScaleNet does not require an explicit search for every possible model size. By identifying the scaling trends within the initial stages (S0 through S3), the framework utilizes theoretically derived regression functions to extrapolate these rules. This allows for the generation of even larger models (S4 and S5) that maintain optimal performance without the need for additional search costs.

Experimental Results

The researchers evaluated ScaleNet primarily on the ImageNet-1k dataset, with additional ablation studies on ImageNet-100 and transfer learning benchmarks.

Performance and Efficiency

ScaleNet models consistently outperformed established benchmarks. Notably, ScaleNet-S0 achieved 77.5% Top-1 accuracy on ImageNet using only 0.35G FLOPs, surpassing EfficientNet-B0 (76.3%). Beyond accuracy, the framework demonstrated remarkable search efficiency. ScaleNet reduced search costs by 2.53× compared to BigNAS and by a factor of 436× compared to MnasNet.

Transfer Learning

To test the robustness of the discovered architectures, the models were fine-tuned on several datasets, including CIFAR-10, CIFAR-100, Stanford Cars, and Food-101. ScaleNet models consistently outperformed Inception-v4 and EfficientNet-B3, achieving higher accuracy while utilizing significantly fewer parameters and FLOPs.

Discussion and Future Work

The core insight of ScaleNet is that scaling is not a universal law but is architecture-dependent. By identifying architectures with high "scaling potential," ScaleNet avoids the performance plateaus often hit by manually scaled models.

Despite its success, the framework currently relies on FLOPs as the primary budget metric. Because theoretical computations do not always correlate perfectly with real-world latency on specific hardware (e.g., mobile GPUs vs. DSPs), future iterations could incorporate hardware-aware constraints into the MCEA search. Additionally, while the current search space is focused on MBConv (Mobile Inverted Bottleneck) blocks, extending this joint scaling approach to Vision Transformers (ViTs) represents a promising avenue for future research.

Summary updated

BibTeX
@inproceedings{xie2022scalenet,
  author = {Jiyang Xie and Xiu Su and Shan You and Zhanyu Ma and Fei Wang and Chen Qian},
  title = {{ScaleNet: Searching for the Model to Scale}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19803-8\_7},
}
Figure