The Ramifications of Making Deep Neural Networks Compact

Nandan Kumar Jha, Sparsh Mittal, Govardhan Mattela

TL;DR

The push for "compact" Deep Neural Networks (DNNs) assumes that reducing parameter counts and multiply-accumulate (MAC) operations naturally leads to better energy efficiency and lower memory usage. However, this paper reveals a counterintuitive truth: parameter-reduction techniques often drastically increase the number of intermediate activations, leading to larger memory footprints, memory bandwidth bottlenecks, and ultimately, worse energy efficiency.

What Problem Does This Paper Address?

To deploy powerful DNNs on resource-constrained edge and mobile devices, researchers have focused on designing "compact" models (like MobileNet, SqueezeNet, and DenseNet). The standard practice is to use the number of parameters and the number of MACs as proxies for a model's memory footprint and energy consumption, respectively.

The authors argue that these proxies are fundamentally flawed. The total memory footprint of a DNN depends not just on weights (parameters), but also on the intermediate feature maps (activations) generated during inference. Furthermore, the energy cost of a MAC operation isn't static; it depends heavily on where the data is fetched from (cache vs. main memory) and how often that data is reused. The paper investigates whether state-of-the-art compact models actually deliver the hardware efficiency they promise.

Methods

The researchers empirically profiled several popular compact DNNs (SqueezeNet, SqueezeNext, MobileNet, DenseNet, GoogLeNet, Inception-V2) alongside a traditional baseline (AlexNet).

  • Hardware Setup: They ran inference using the Caffe framework on a high-end NVIDIA Tesla P100 GPU. Using a powerful GPU ensured that hardware limitations didn't mask the inherent architectural bottlenecks of the DNNs.
  • Profiling: They used the NVIDIA Visual Profiler to perform kernel-level analysis, measuring memory footprint, inference time, throughput (FPS), and energy efficiency (MACs/Joule).
  • Novel Metrics: Because measuring true "arithmetic intensity" (compute vs. memory access) is difficult, the authors propose using the MACs/parameter ratio and MACs/activation ratio as proxies. They also propose the activations/parameters ratio as a measure of a network's non-linearity.

Datasets

Not explicitly discussed in the paper. The authors focus on hardware profiling during inference using standard input image dimensions (e.g., $224 \times 224$ or $231 \times 231$) rather than evaluating task accuracy on a specific dataset like ImageNet.

Key Insights and Ideas

  • The Factorization Trap: A common trick to reduce parameters is filter factorization (e.g., replacing one $5 \times 5$ filter with two $3 \times 3$ filters). While this cuts parameters and MACs in half, it doubles the number of output activations.
  • Activations Drive Memory Footprint: The authors found a very high correlation (Pearson coefficient of 0.82) between the sum of parameters + activations and the total memory footprint. Because compact models aggressively reduce parameters, activations end up dominating the memory footprint.
  • $1 \times 1$ Convolutions are Inefficient: Models like MobileNet and SqueezeNet rely heavily on $1 \times 1$ convolutions. While computationally cheap, they offer terrible data reuse. This results in low arithmetic intensity, meaning the GPU spends most of its time waiting for data to be fetched from memory rather than doing actual math.
  • Dense Connections Hog Memory: Architectures with complex routing, like DenseNet, require multiple copies of the same activation maps to be kept alive in memory simultaneously, causing memory requirements to skyrocket, especially as batch sizes increase.

Key Results

  • AlexNet is the most energy-efficient: Despite having a massive 61M parameters, AlexNet achieved the highest energy efficiency (75.94 G-MACs/Joule). This is because it has a very high MACs/activation ratio (352.65), meaning data fetched from memory is reused heavily.
  • MobileNet is the least energy-efficient: MobileNet achieved only 1.65 G-MACs/Joule and the lowest throughput (42 FPS). Kernel analysis showed that 86% of its inference time was spent in memory-bound, sparse-matrix style operations with poor Streaming Multiprocessor (SM) utilization.
  • SqueezeNext's memory bloat: A variant of SqueezeNext (1.0-G-SqNxt-23) has 112x fewer parameters than AlexNet. Yet, at a batch size of 1, it actually consumes more memory (1019 MB vs 1015 MB). Its activations-to-parameters ratio is a staggering 32.8, compared to AlexNet's 0.03.
  • DenseNet's footprint: DenseNet-121 consumed the most memory of any tested model (1405 MB at batch size 1) due to its dense connectivity.

Limitations and Future Work

Limitations: The profiling was conducted exclusively on a high-end data center GPU (Tesla P100). While this is excellent for isolating the mathematical bottlenecks of the algorithms, actual edge devices (mobile phones, IoT sensors) have vastly different memory hierarchies, caches, and compute architectures, which might respond differently to these workloads.

Future Work: The authors suggest that future neural network design must focus on increasing arithmetic intensity—specifically by optimizing the MACs/activation ratio—rather than just blindly reducing parameter counts. They plan to extend this profiling to different hardware platforms (like FPGAs and ReRAM) and develop design-time tools to help researchers estimate the true energy efficiency of a model before training it.

Summary updated

Abstract

The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory-footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the " memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.
BibTeX
@inproceedings{jha2019ramifications,
  author = {Nandan Kumar Jha and Sparsh Mittal and Govardhan Mattela},
  title = {{The Ramifications of Making Deep Neural Networks Compact}},
  booktitle = {VLSID},
  year = {2019},
  url = {https://openreview.net/forum?id=HKSMal7qYK},
}

Loading related content...

Figure