Bit-Pragmatic Deep Neural Network Computing

Jorge Albericio, Patrick Judd, Alberto Delmas, Sayeh Sharify, Andreas MoshovosICLR 2017 · OpenReview

Introduction

Deep Neural Networks (DNNs) require massive computational resources, primarily driven by the multiplication operations in convolutional layers. Pragmatic (PRA) is a novel hardware architecture designed to accelerate DNN inference by skipping "ineffectual" computations at the bit level. By processing only the non-zero bits of activations, PRA achieves a massive 4.3x to 4.5x speedup over state-of-the-art accelerators like DaDianNao, while significantly improving energy efficiency.

The Problem: Wasted Cycles in Standard Multipliers

Standard DNN hardware typically relies on bit-parallel multipliers. When multiplying two numbers—such as a weight and an activation—the hardware processes every bit, regardless of whether it is a 1 or a 0. The authors identify two major sources of waste in this conventional approach:

  1. Excess of Precision (EoP): Hardware is often fixed at a specific width (e.g., 16 bits), even if the actual value only requires 5 bits to be represented accurately. The extra "prefix" and "suffix" bits result in wasted compute cycles.
  2. Lack of Explicitness (LoE): In standard positional binary notation, many bits are zero. Multiplying by a zero bit adds nothing to the final sum, yet standard multipliers expend time and energy processing these operations anyway.
Essential bits and precision in hardware

In modern image classification networks, the authors found that a staggering 93% of activation bits are ineffectual in 16-bit fixed-point representations. Even in 8-bit quantized models, 69% of bits are zeros. Consequently, current hardware spends the vast majority of its operating time performing zero-value multiplications.

The Pragmatic Architecture (PRA)

To address these inefficiencies, the authors propose the Pragmatic (PRA) architecture, which calculates only the non-zero terms of a multiplication.

Comparison of a standard Bit-Parallel Unit and the proposed Pragmatic Unit

Key innovations of the PRA design include:

  • Bit-Serial Activation Processing: Instead of processing all bits of an activation simultaneously (bit-parallel), PRA processes them serially, but only targets the bits that are "1". This makes the execution time proportional to the number of non-zero bits rather than the total bit-width.
  • On-the-Fly Conversion: Activations are stored in standard formats in memory to save space. As they are broadcast to the compute units, they are converted into an "explicit representation"—a list of offsets for non-zero bits.
  • Judicious SIMD Grouping: To maintain high throughput and wide memory access, PRA groups compute lanes together. It uses "pallet-level" synchronization, where a group of units waits for the slowest activation (the one with the most 1-bits) before moving to the next set of data.
  • 2-Stage Shifting: To keep the hardware footprint small, large bit-shifts are decomposed into two smaller stages, reducing the area required for the shifter and adder trees.
  • Software-Guided Trimming: The architecture allows software to specify how many prefix and suffix bits can be safely ignored for each layer, further reducing the "essential bit content" without sacrificing model accuracy.

Evaluation and Key Results

The authors evaluated PRA using several well-known Convolutional Neural Networks (CNNs) for image classification, including AlexNet, NiN (Network in Network), GoogLeNet, and VGG (Models M, S, and 19). These models were tested using both 16-bit fixed-point and 8-bit quantized representations.

The evaluation yielded several compelling insights:

  • Massive Speedups: PRA achieves an average speedup of 4.3x over the DaDianNao (DaDN) baseline for 16-bit models and 4.5x for 8-bit quantized models. Execution time in PRA is highly data-dependent; layers with sparser activations run significantly faster.
  • Energy Efficiency: The architecture is 1.48x more energy-efficient than DaDN. While the logic for bit-serial processing is inherently more complex, the massive reduction in total cycles more than compensates for the increased power per cycle.
  • Modest Area Overhead: These performance and efficiency gains come at a relatively low cost, with only a 1.35x area overhead compared to the baseline architecture.
  • Quantization Synergy: PRA works exceptionally well with 8-bit quantization, proving that bit-pragmatic computing remains a highly viable strategy even as the industry shifts toward lower-precision models.

Limitations and Future Work

While highly effective, the current PRA implementation specifically targets ineffectual bits in activations. The authors note that network weights also contain many zero bits, presenting a clear target for future optimization. Additionally, the paper focuses strictly on inference; applying these bit-pragmatic concepts to training would be significantly more complex due to the need for high-precision gradients. Finally, while convolutional layers dominate execution time in CNNs, future work could explore how PRA handles Recurrent Neural Networks (RNNs) or fully connected layers more broadly.

Summary updated

Abstract

We quantify a source of ineffectual computations when processing the multiplications of the convolutional layers in Deep Neural Networks (DNNs) and propose Pragrmatic (PRA), an architecture that exploits it improving performance and energy efficiency. The source of these ineffectual computations is best understood in the context of conventional multipliers which generate internally multiple terms, that is, products of the multiplicand and powers of two, which added together produce the final product. At runtime, many of these terms are zero as they are generated when the multiplicand is combined with the zero-bits of the multiplicator. While conventional bit-parallel multipliers calculate all terms in parallel to reduce individual product latency, PRA calculates only the non-zero terms resulting in a design whose execution time for convolutional layers is ideally proportional to the number of activation bits that are 1. Measurements demonstrate that for the convolutional layers on Convolutional Neural Networks and during inference, PRA improves performance by 4.3x over the DaDiaNao (DaDN) accelerator and by 4.5x when DaDN uses an 8-bit quantized representation. DaDN was reported to be 300x faster than commodity graphics processors.
BibTeX
@inproceedings{albericio2017bit,
  author = {Jorge Albericio and Patrick Judd and Alberto Delmas and Sayeh Sharify and Andreas Moshovos},
  title = {{Bit-Pragmatic Deep Neural Network Computing}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=By14kuqxx},
}
Figure