Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip

NeurIPS 1990

TL;DR

This paper details the implementation of the Back Propagation (BP) algorithm on the Adaptive Solutions CNAPS neurocomputer chip, a novel parallel architecture. The CNAPS chip demonstrates impressive performance, achieving 2.3 billion training weight updates per second and 9.6 billion feed-forward connections per second with an 8-chip configuration, showcasing its potential for accelerating neural network computations.

What Problem Does This Paper Address?

Neural networks, particularly for tasks like machine learning, demand significant computational power due to their inherent parallelism and complex calculations. Traditional approaches often rely on large parallel computers or specialized chips limited to a narrow set of algorithms. This paper addresses the need for a general-purpose, highly parallel neurocomputer architecture capable of efficiently implementing a broad range of current neural network algorithms, specifically focusing on the widely used Back Propagation algorithm. The challenge lies in adapting BP, which typically uses floating-point arithmetic, to a hardware architecture optimized for integer operations while maintaining accuracy and achieving high performance.

Methods

The core of this paper is the implementation of the Back Propagation algorithm on the Adaptive Solutions CNAPS neurocomputer chip. The CNAPS architecture features a 64-processor chip, each with local memory and running at 25 MHz, designed for parallel execution. Key aspects of the implementation include:

  • On-Chip Learning: The CNAPS chip supports on-chip learning, meaning weight updates can occur directly on the hardware, reducing the need to transfer data back and forth to external memory.
  • Integer Arithmetic: To achieve high performance and efficiency, the implementation leverages fixed-point integer arithmetic rather than floating-point. Weights are represented using 16 bits (4 bits for the integer part, including sign, and 12 bits for the fractional part), allowing for a range of -8 to +8. Input and output values are 8-bit unsigned integers, and the learning rate is an 8-bit integer.
  • Weight Transposition: A common technique in BP implementation, the paper describes how the transpose of the output layer weights is stored on hidden node processors to facilitate error back-propagation.
  • Sigmoid Function Approximation: The sigmoid output function, crucial for neural network activation, is implemented using an 8-bit by 256 lookup table for efficient computation.
  • Hardware Simulator: Performance figures are derived from a clock-accurate hardware simulator, providing a realistic estimate of the chip's capabilities.

Datasets

Not discussed in the paper. The paper focuses on the implementation and performance of the BP algorithm on the CNAPS hardware, using a benchmark network configuration for evaluation rather than specific datasets.

Key Insights and Ideas

  • Efficient Integer Arithmetic for BP: The paper demonstrates that 16-bit fixed-point integer arithmetic is sufficient for training benchmark BP applications with results comparable to floating-point versions, a crucial insight for hardware acceleration.
  • High Parallelism for BP: The CNAPS architecture's parallel design, with its multiple processors and efficient inter-processor communication, is well-suited for the inherently parallel nature of the BP algorithm.
  • Integrated Learning and Processing: The ability to perform on-chip learning significantly reduces data transfer bottlenecks, a major advantage for accelerating neural network training.
  • Versatility of the CNAPS Architecture: The paper highlights that the CNAPS chip is a general-purpose neurocomputer, capable of implementing most current neural network algorithms, not just BP.

Key Results

An 8-chip configuration of the Adaptive Solutions CNAPS neurocomputer (totaling 512 processors) achieved the following performance figures on a benchmark network with 1900 inputs, 500 hidden nodes, and 12 outputs:

  • Training Phase (Weight Updates): 2.3 billion connections per second.
  • Feed Forward Phase (Connections Processed): 9.6 billion connections per second.

These results are presented in comparison to other high-performance computing systems, showing the CNAPS chip to be highly competitive, especially considering its specialized architecture and 16-bit integer representation.

Limitations and Future Work

  • Limitations: The paper acknowledges that the implementation uses a specific fixed-point representation for weights and activations. While comparable to floating-point results for the benchmark, the generalizability of this precision across all BP applications might require further investigation. The performance figures are based on a hardware simulator, and real-world performance might vary.
  • Future Work: The paper suggests that the CNAPS architecture's ability to map multiple networks simultaneously and use the output of one as input to another opens up possibilities for complex, multi-stage neural network computations without intermediate data transfers. This could be applied to areas like speech recognition, where Fourier transforms can be directly fed into subsequent BP networks.

Summary updated

BibTeX
@inproceedings{1990back,
  title = {{Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure