Abstract
In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about O(s) increase in model activation w.r.t. scaling flops by a factor of s, the proposed fast compound scaling results in close to O(sqrt s ) increase in activations, while achieving excellent accuracy. Fewer activations leads to speedups on modern memory-bandwidth limited hardware (e.g., GPUs). More generally, we hope this work provides a framework for analyzing scaling strategies under various computational constraints.
The Problem: FLOPs are Not Runtime
In the quest for higher accuracy, researchers often take a successful "base" model and scale it up. The industry standard has been Compound Scaling (popularized by EfficientNet), which scales a model's depth ($d$), width ($w$), and resolution ($r$) in equal proportions.
However, these scaling strategies are almost exclusively optimized for the trade-off between accuracy and FLOPs (Floating Point Operations). On modern, highly parallel hardware like GPUs and TPUs, FLOPs are a poor proxy for actual wall-clock time. The real bottleneck is often memory bandwidth—how fast data can be moved—rather than how many math operations can be performed. This paper investigates scaling strategies that optimize for both accuracy and actual model runtime by focusing on activations.
Complexity Analysis and the Activation Proxy
The authors provide a rigorous complexity analysis of how different scaling dimensions affect three metrics: FLOPs ($f$), Parameters ($p$), and Activations ($a$). They demonstrate that for a fixed increase in FLOPs (by a factor of $s$), scaling depth or resolution increases activations by $O(s)$. Crucially, scaling width only increases activations by $O(\sqrt{s})$.
As shown in the figure above, while different scaling strategies (compound, depth+width, fast, and width-only) result in very similar error rates for a given FLOP count, their impact on activations and actual epoch time is vastly different. The "compound" strategy, which scales depth and resolution significantly, leads to the highest activation growth and the slowest runtimes.
Fast Compound Scaling ($dWr$)
Based on these insights, the authors introduce Fast Compound Scaling. While traditional uniform compound scaling treats $d, w,$ and $r$ equally, fast scaling primarily increases width while allowing for modest increases in depth and resolution to maintain high accuracy. This strategy results in a growth in activations close to $O(\sqrt{s})$, leading to significant speedups on memory-bandwidth limited hardware.
Key Results and Hardware Efficiency
Through extensive benchmarking, the authors show that activations have a near-perfect correlation ($r \geq 0.99$) with runtime on GPUs, whereas FLOPs are much less predictive ($r \approx 0.81$).
As illustrated in the correlation plot, for a fixed FLOP count (e.g., 30B), the actual epoch time can vary by a factor of 3x depending on the scaling strategy used. Models scaled using the $dWr$ (fast scaling) strategy are nearly as fast as width-only scaled models but achieve significantly higher accuracy, matching the performance of much slower compound-scaled models.
For example, a RegNetY model scaled to 16GF using fast scaling is actually faster and more accurate than an EfficientNet-B4, despite the RegNetY model having nearly $4\times$ more FLOPs. This highlights how misleading FLOPs can be as a performance metric and underscores the importance of considering memory-related metrics like activations in model design.
Conclusion
By prioritizing width scaling over depth and resolution, the "fast scaling" strategy produces models that match state-of-the-art accuracy while being significantly faster and more memory-efficient on GPUs. This work provides a new framework for analyzing scaling strategies, shifting the focus from theoretical operation counts to practical hardware performance.
Summary updated