HarDNet: A Low Memory Traffic Network

Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, Youn-Long LinICCV 2019 · DOI · arXiv

In the field of deep learning, performance is traditionally measured by MACs (Multiply-Accumulate operations) and Parameter Count. The prevailing assumption is that reducing these metrics directly translates to faster inference. However, many "lightweight" models struggle on edge devices due to the "Memory Wall."

HarDNet (Harmonic Densely Connected Network) challenges this paradigm, arguing that memory traffic—the data transfer between the chip and DRAM—is the primary bottleneck for inference speed, particularly in high-resolution tasks like video segmentation.

The Problem: MACs Do Not Equal Latency

Modern hardware, including GPUs and AI accelerators, often spends more time moving intermediate feature maps than performing calculations. Architectures like DenseNet, while parameter-efficient, create significant memory traffic because they require numerous small tensors to remain "alive" in memory for long periods. To quantify this, the researchers introduced two hardware-independent metrics:

  1. CIO (Convolutional Input/Output): The sum of the sizes of all input and output tensors, serving as a proxy for DRAM traffic.
  2. MoC (MACs over CIO): Referred to as "Computational Density." A high MoC indicates efficient hardware utilization, while a low MoC suggests the processor is idling while waiting for data.
Computational density (MoC) as a function of channel width and convolution type. High MoC indicates better hardware utilization.

The HarDNet Architecture

HarDNet addresses the memory traffic issue through a "harmonic" connection pattern inspired by power-of-two waves. Unlike DenseNet's dense connectivity, HarDNet employs a specific sparsification strategy: Layer $k$ connects to layer $k-2^n$ only if $2^n$ divides $k$.

Key Design Principles:

  • Sparsification: This power-of-two scheme allows the system to flush intermediate layers from the cache sooner, drastically reducing the CIO.
  • Channel Weighting: To compensate for fewer connections, the network increases the channel width of "influential" layers (those divisible by larger powers of two), ensuring robust feature learning.
  • Inverted Transition Down: Transition modules were redesigned to reduce CIO by 50% compared to standard DenseNet blocks.
Comparison of connection patterns between DenseNet, LogDenseNet/SparseNet, and the proposed HarDNet. HarDNet uses a harmonic pattern to reduce memory traffic.

Experimental Results

The authors validated HarDNet across several benchmarks, including ImageNet classification, CamVid semantic segmentation, and object detection (PASCAL VOC/MS COCO).

  • Inference Speed: On the CamVid dataset, HarDNet achieved accuracy comparable to FC-DenseNet103 but with a 35% reduction in inference time.
  • Classification Efficiency: HarDNet-68 outperformed ResNet-101 on ImageNet while being significantly faster and maintaining a smaller footprint.
  • Edge Deployment: As a backbone for SSD, HarDNet-68 surpassed both VGG-16 and ResNet-101 in mean Average Precision (mAP), proving highly competitive for real-time applications on devices like the Nvidia Jetson Nano.

Conclusion and Future Work

HarDNet demonstrates that inference latency is more closely correlated with memory traffic (CIO) than with raw operation counts. While highly effective for mid-to-large scale networks, the authors note that very small models may see diminishing returns as layer overhead begins to outweigh traffic savings. Future research may involve refining the CIO metric to account for compiler-level optimizations like layer fusion and further exploring the relationship between CIO and depth-wise separable convolutions.

Summary updated

BibTeX
@inproceedings{chao2019hardnet,
  author = {Ping Chao and Chao-Yang Kao and Yu-Shan Ruan and Chien-Hsiang Huang and Youn-Long Lin},
  title = {{HarDNet: A Low Memory Traffic Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00365},
}
Figure