Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

Maurizio Capra, Beatrice Bussolino, Alberto Marchisio, Guido Masera, Maurizio Martina, Muhammad Shafique

TL;DR

This comprehensive survey explores the multi-layered landscape of Deep Neural Network (DNN) and Spiking Neural Network (SNN) optimization. It highlights how the synergy between hardware-aware software (quantization, pruning) and specialized spatial architectures (FPGAs, ASICs) is the only viable path toward deploying sophisticated AI on power-constrained edge devices.

What Problem Does This Paper Address?

Deep Learning has revolutionized fields from medical imaging to autonomous driving, but its "intelligence" comes at a high cost. Modern DNNs require billions of Multiply-and-Accumulate (MAC) operations for a single inference, making them incredibly compute- and energy-hungry.

The authors address the "Von Neumann Bottleneck"—the energy-intensive process of moving data between memory and processors—and the mismatch between general-purpose hardware (CPUs/GPUs) and the specific needs of neural networks. As we move toward "AI at the Edge" (IoT, wearables, and robotics), we need systems that provide high performance within a strict power envelope.

Methods

As a survey paper, the authors synthesize a vast array of techniques across the entire computing stack:

  • Hardware Architectures: They distinguish between Temporal Architectures (CPUs/GPUs that use SIMD/SIMT to process data sequentially) and Spatial Architectures (FPGAs/ASICs that use arrays of Processing Elements to move data directly between units).
  • Dataflow Optimization: The paper analyzes four primary dataflows—Weight Stationary, Output Stationary, Row Stationary, and No Local Reuse—explaining how each minimizes expensive DRAM accesses by maximizing data reuse within local buffers.
  • Algorithmic Compression:
    • Quantization: Moving from 32-bit floating-point to 8-bit, 1-bit (Binary), or Ternary representations.
  • Pruning: Removing redundant weights, neurons, or entire channels to create "sparse" networks.
  • Knowledge Distillation: Training a small "student" model to mimic a large, complex "teacher" model.
  • Hardware-Aware Neural Architecture Search (NAS): Using AI to automatically design network structures that are optimized for specific hardware targets.
  • Spiking Neural Networks (SNNs): Exploring the "third generation" of neural networks that use bio-inspired, event-driven spikes to achieve extreme energy efficiency.

Datasets

The paper provides a comprehensive summary of the benchmarks used to evaluate these optimizations:

  • Classic Image Datasets: MNIST (handwritten digits), CIFAR-10/100, and the massive ImageNet (1.3M images).
  • Object Detection/Segmentation: COCO (Common Objects in Context) and Open Images V6.
  • Specialized Benchmarks: CORe50 (for continual object recognition) and ObjectNet (designed to test model robustness against rotation and background changes).

Key Insights and Ideas

  • Memory is the Real Enemy: A DRAM access costs roughly two orders of magnitude more energy than a MAC operation. Therefore, the most successful optimizations aren't just about faster math; they are about reducing data movement.
  • The Power of Sparsity: Between ReLU activations (which zero out negative values) and pruning, many neural network matrices are 50-80% empty. Specialized hardware that can "skip" these zeros (zero-skipping) sees massive gains in speed and efficiency.
  • Co-Design is Mandatory: You cannot optimize the software and hardware in isolation. Hardware-aware NAS is becoming essential because a model that is "fast" on a GPU might be "slow" on an FPGA due to different memory hierarchies.
  • SNNs are the Wild Card: While SNNs offer the promise of brain-like efficiency, they are difficult to train because their functions are non-differentiable. The "road ahead" involves hybrid training methods that convert DNNs to SNNs.

Key Results

  • Quantization Gains: The authors note that 8-bit dynamic fixed-point quantization can achieve <1% accuracy loss while being 20x more energy-efficient than standard 32-bit floating-point operations.
  • Pruning Efficiency: Techniques like "Deep Compression" can reduce the number of parameters in models like AlexNet by 10x without significant accuracy degradation.
  • Hardware Comparison: While GPUs lead in raw throughput (GFLOPS), ASICs and FPGAs provide significantly better GFLOPS/WATT, making them the preferred choice for edge deployment.
  • The Memory Wall: The paper highlights High Bandwidth Memory (HBM) and In-Memory Computing (IMC) as the most promising hardware solutions to overcome the bandwidth limits of traditional DDR memory.

Limitations and Future Work

  • The AGI Gap: Despite these optimizations, we are still far from Artificial General Intelligence (AGI). Current hardware and models are still largely "narrow" and specialized.
  • Toolchain Fragmentation: There is a lack of a unified programming language or compiler that can seamlessly target the diverse landscape of AI accelerators (CPUs, GPUs, FPGAs, and ASICs).
  • Security Vulnerabilities: The authors acknowledge that optimizations like quantization can sometimes make models more susceptible to adversarial attacks. Future work must integrate security into the hardware-software co-design process.
  • SNN Training: Bridging the accuracy gap between SNNs and DNNs remains a primary research challenge for the neuromorphic community.

Summary updated

Abstract

Currently, Machine Learning (ML) is becoming ubiquitous in everyday life. Deep Learning (DL) is already present in many applications ranging from computer vision for medicine to autonomous driving of modern cars as well as other sectors in security, healthcare, and finance. However, to achieve impressive performance, these algorithms employ very deep networks, requiring a significant computational power, both during the training and inference time. A single inference of a DL model may require billions of multiply-and-accumulated operations, making the DL extremely compute- and energy-hungry. In a scenario where several sophisticated algorithms need to be executed with limited energy and low latency, the need for cost-effective hardware platforms capable of implementing energy-efficient DL execution arises. This paper first introduces the key properties of two brain-inspired models like Deep Neural Network (DNN), and Spiking Neural Network (SNN), and then analyzes techniques to produce efficient and high-performance designs. This work summarizes and compares the works for four leading platforms for the execution of algorithms such as CPU, GPU, FPGA and ASIC describing the main solutions of the state-of-the-art, giving much prominence to the last two solutions since they offer greater design flexibility and bear the potential of high energy-efficiency, especially for the inference process. In addition to hardware solutions, this paper discusses some of the important security issues that these DNN and SNN models may have during their execution, and offers a comprehensive section on benchmarking, explaining how to assess the quality of different networks and hardware systems designed for them.
BibTeX
@article{capra2020hardware,
  author = {Maurizio Capra and Beatrice Bussolino and Alberto Marchisio and Guido Masera and Maurizio Martina and Muhammad Shafique},
  title = {{Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3039858},
}

Loading related content...

Figure