Abstract
The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the "memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.
Introduction
The push for "compact" Deep Neural Networks (DNNs) assumes that reducing parameter counts and multiply-accumulate (MAC) operations naturally leads to better energy efficiency and lower memory usage. To deploy powerful DNNs on resource-constrained edge and mobile devices, researchers have focused on designing models like MobileNet, SqueezeNet, and DenseNet. The standard practice is to use the number of parameters and the number of MACs as proxies for a model's memory footprint and energy consumption, respectively.
However, these proxies are fundamentally flawed. The total memory footprint of a DNN depends not just on weights (parameters), but also on the intermediate feature maps (activations) generated during inference. Furthermore, the energy cost of a MAC operation depends heavily on data locality and reuse. This paper investigates whether state-of-the-art compact models actually deliver the hardware efficiency they promise.
Methodology
The researchers empirically profiled several popular compact DNNs (SqueezeNet, SqueezeNext, MobileNet, DenseNet, GoogLeNet, Inception-V2) alongside a traditional baseline (AlexNet). Inference was performed using the Caffe framework on an NVIDIA Tesla P100 GPU. The NVIDIA Visual Profiler was used for kernel-level analysis, measuring memory footprint, inference time, throughput (FPS), and energy efficiency (MACs/Joule). To better understand architectural bottlenecks, the authors proposed novel metrics: the MACs/parameter ratio, MACs/activation ratio, and the activations/parameters ratio.
The Factorization Trap and Memory Bloat
A common technique to reduce parameters is filter factorization (e.g., replacing one $5 \times 5$ filter with two $3 \times 3$ filters). While this reduces parameters and MACs, it often doubles the number of output activations. Because compact models aggressively reduce parameters, activations end up dominating the memory footprint. The authors found a high correlation (0.82) between the sum of parameters and activations and the total memory footprint.
Architectures with complex routing, like DenseNet, require multiple copies of activation maps to be kept alive simultaneously, causing memory requirements to skyrocket. For instance, a variant of SqueezeNext (1.0-G-SqNxt-23) has 112x fewer parameters than AlexNet but consumes more memory at a batch size of 1 due to an activations-to-parameters ratio of 32.8.
Comparative Performance Analysis
Counterintuitively, AlexNet—despite its 61M parameters—was the most energy-efficient model tested, achieving 75.94 G-MACs/Joule. This efficiency stems from a high MACs/activation ratio (352.65), indicating heavy data reuse. In contrast, MobileNet was the least energy-efficient (1.65 G-MACs/Joule) and had the lowest throughput. Kernel analysis revealed that 86% of MobileNet's inference time was spent in memory-bound operations with poor Streaming Multiprocessor (SM) utilization.
Models relying heavily on $1 \times 1$ convolutions, like MobileNet and SqueezeNet, suffer from low arithmetic intensity. While computationally cheap, these operations offer poor data reuse, forcing the GPU to wait for data fetches from memory.
Conclusion
The study demonstrates that blindly reducing parameter counts can lead to unintended consequences, such as increased activation counts and reduced arithmetic intensity. Future neural network design should focus on optimizing the MACs/activation ratio to improve true energy efficiency on hardware. While this profiling was conducted on a high-end GPU, the fundamental mathematical bottlenecks identified suggest that compact models may face similar challenges on edge devices with more constrained memory hierarchies.
Summary updated