Abstract: In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about $O(s)$ increase in model activation w.r.t. scaling flops by a factor of $s$, the proposed fast compound scaling results in close to $O(\sqrt{s})$ increase in activations, while achieving excellent accuracy. This leads to comparable speedups on modern memory-limited hardware (e.g., GPU, TPU). More generally, we hope this work provides a framework for analyzing and selecting scaling strategies under various computational constraints.
Introduction
Scaling convolutional neural networks (CNNs) is a fundamental technique for improving model performance. By increasing a base model's width, depth, or input resolution, researchers can create larger models with greater representational power. Since the introduction of EfficientNet, "compound scaling"—the simultaneous scaling of all three dimensions—has become the industry standard.
However, the primary metric for evaluating these strategies has traditionally been FLOPs (Floating Point Operations). This paper from Facebook AI Research (FAIR) argues that FLOPs are a deceptive metric for modern hardware. On highly parallel processors like GPUs and TPUs, the bottleneck is often memory bandwidth rather than raw computation. Consequently, models that appear efficient on paper due to low FLOP counts can be significantly slower in practice than models with higher FLOPs but lower memory overhead.
The Scaling Complexity Framework
To understand these trade-offs, the authors developed a framework to analyze how scaling dimensions affect three key metrics: FLOPs ($f$), parameters ($p$), and activations ($a$). They discovered a critical disparity in how different scaling strategies impact model activations.
As shown in Figure 1, for a scale factor $s$, scaling width ($w$) increases activations by only $O(\sqrt{s})$. In contrast, scaling depth ($d$) or resolution ($r$) increases them by $O(s)$. This "activation tax" is the primary driver of increased runtime on modern hardware. By prioritizing width scaling, models can achieve similar accuracy with significantly lower activation counts, leading to faster training and inference.
The Activation Insight
The core contribution of this work is the empirical proof that activations, not FLOPs, are the best predictor of model runtime. The authors analyzed the correlation between various metrics and actual wall-clock time (epoch time) across multiple model families.
As illustrated in Figure 5, the correlation between activations and runtime is nearly perfect ($r \ge 0.99$) within a model family. FLOPs are significantly less predictive ($r \approx 0.81$), and parameters are even less so ($r \approx 0.56$). This confirms that most models used in practice are "memory bound" rather than "flop bound."
Fast Compound Scaling ($dW r$)
Based on these insights, the authors propose Fast Compound Scaling ($dW r$). This strategy uses a parameter $\alpha$ to control the relative scaling of width versus other dimensions. While uniform compound scaling (like EfficientNet) uses $\alpha = 1/3$, the authors identify $\alpha = 0.8$ as the optimal "Fast Scaling" point. This strategy primarily increases width while allowing for modest increases in depth and resolution.
Key results from applying Fast Scaling include:
- Superior Speed-Accuracy Trade-offs: A RegNetY-16GF model using Fast Scaling was found to be both faster and more accurate than EfficientNet-B4, despite having 4x the FLOPs.
- Consistency Across Architectures: Fast Scaling consistently matched the accuracy of uniform compound scaling across EfficientNet, RegNetY, and RegNetZ, while being significantly faster.
- Memory Efficiency: Lower activation counts allow for larger batch sizes during training, further improving hardware utilization.
Conclusion
This work shifts the focus of model scaling from theoretical compute (FLOPs) to practical runtime constraints (activations). By adopting Fast Compound Scaling, researchers can design models that are optimized for the hardware they actually run on. Future work may explore redistributing blocks across network stages or applying these scaling principles to tasks like object detection and segmentation, where resolution trade-offs are more complex.
Summary updated