Neural acceleration for general-purpose approximate programs

Hadi Esmaeilzadeh, Adrian Sampson, Luis Ceze, Doug BurgerCommunications of the ACM 2015 · DOI

TL;DR

This paper introduces "Parrot," a novel program transformation that replaces sections of approximate-tolerant imperative code with neural networks (NNs) executed on a dedicated Neural Processing Unit (NPU). This approach achieves significant performance and energy savings (2.3x speedup and 3.0x energy reduction on average) by leveraging hardware acceleration for NNs, opening up new avenues for optimizing general-purpose applications that can tolerate minor inaccuracies.

What Problem Does This Paper Address?

The paper tackles the growing challenge of energy efficiency in computing. As Dennard scaling has slowed, traditional performance gains from general-purpose processors have diminished. While ASICs offer high efficiency, they lack flexibility for rapidly evolving software. Programmable accelerators like GPUs and FPGAs offer a middle ground, but many modern applications, such as image processing, data mining, and augmented reality, can tolerate some level of imprecision in their computations. This tolerance presents a significant opportunity for performance and energy gains that current architectures don't fully exploit. The paper aims to bridge this gap by enabling the acceleration of these "approximate-tolerant" code regions using specialized hardware.

Methods

The core innovation is the Parrot transformation, a three-phase process:

  1. Programming: Programmers identify "hot" (frequently executed), approximable code regions with well-defined inputs and outputs. This is done via simple annotations.
  2. Compilation:
    • Code Observation: The compiler instruments the annotated code to collect input-output pairs during execution with representative test inputs. This data forms the training and validation sets.
  3. Neural Network Training: The compiler automatically selects and trains a suitable neural network (specifically, a multi-layer perceptron or MLP) to mimic the behavior of the original code region. This involves a topology search to balance accuracy and efficiency, and backpropagation to learn synaptic weights.
  4. Code Generation: The compiler generates NPU configuration data (topology, weights) and replaces the original code with special instructions to invoke the NPU.
  5. Execution: The transformed program runs on an NPU-augmented core. When an annotated code region is encountered, the NPU executes the trained neural network, offloading computation from the main CPU.

A key architectural component is the Neural Processing Unit (NPU), designed as a tightly-coupled, low-latency accelerator. It features multiple processing engines (PEs) and communicates with the CPU via FIFO queues, supporting speculative execution for efficient invocation.

Datasets

The paper does not introduce new benchmark datasets. Instead, it uses existing, diverse applications and their typical input sets for evaluation. For training, representative program inputs (e.g., sample images, random inputs within defined ranges) are used. For evaluation, different input sets (different random inputs or images) are employed to ensure the trained models generalize well.

Key Insights and Ideas

  • Neural Networks as General-Purpose Accelerators: The paper demonstrates that neural networks are not limited to traditional machine learning tasks but can effectively mimic diverse imperative code regions, offering a new paradigm for hardware acceleration.
  • Automatic Transformation for Approximate Computing: The Parrot transformation automates the complex process of mapping approximable code to hardware, abstracting away much of the low-level detail from the programmer.
  • The Importance of Reconfigurability: Different code regions require different neural network topologies. The proposed NPU design is reconfigurable to accommodate this diversity, highlighting that a one-size-fits-all NN architecture is insufficient.
  • Tight Integration is Crucial: The success of NPU acceleration hinges on low-overhead invocation. Tightly coupling the NPU with the processor pipeline and supporting speculative execution are critical for achieving significant speedups, especially for fine-grained code regions.
  • Approximation is a Feature, Not a Bug: By embracing the inherent approximate nature of neural networks, significant performance and energy gains can be realized in applications that can tolerate minor inaccuracies.

Key Results

  • Average Speedup: 2.3x across a suite of diverse applications (FFT, inverse kinematics, triangle intersection, JPEG compression, k-means clustering, Sobel edge detection).
  • Average Energy Savings: 3.0x.
  • Accuracy: Average output quality loss of at most 9.6% across applications, with the majority of output elements exhibiting less than 10% error.
  • NPU Design: An 8-PE reconfigurable digital NPU design was used, showing diminishing returns beyond 8 PEs.
  • Communication Latency Sensitivity: Speedup is sensitive to communication latency, especially for applications with simpler NNs.
  • Software vs. Hardware: Software execution of NNs (using FANN library) resulted in significant slowdowns, underscoring the necessity of dedicated hardware acceleration.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Applicability: While the Parrot transformation works for many approximable code regions, it's not universally applicable. Target code must be "hot," tolerant to imprecision, and have bounded, statically identifiable inputs/outputs. Future work could explore better static analysis to infer suitability.
  • Programmer Effort: Currently, programmers must identify approximable regions and provide training data. Future research could automate these steps, perhaps by inferring approximable blocks or using advanced test generation techniques.
  • Quality and Error Control: While average accuracy is high, the unpredictability of NNs means occasional large errors are possible. Future work should focus on mechanisms to mitigate the frequency of such low-quality results, such as runtime error prediction or on-demand retraining.
  • NPU Implementation: The paper focuses on a digital ASIC NPU. Analog NPUs are identified as having significant potential for even greater efficiency and performance.
  • On-line Training: The current system uses off-line training. Exploring on-line training could improve accuracy but requires careful management of runtime overheads.

Summary updated

BibTeX
@article{esmaeilzadeh2015neural,
  author = {Hadi Esmaeilzadeh and Adrian Sampson and Luis Ceze and Doug Burger},
  title = {{Neural acceleration for general-purpose approximate programs}},
  journal = {Communications of the ACM},
  year = {2015},
  doi = {10.1145/2589750},
}
Figure