CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Yuan-Fang Li, Ardavan Pedram

Abstract

Accelerating the inference of a trained DNN is a well studied subject. In this paper we switch the focus to the training of DNNs. The training phase is compute intensive, demands complicated data communication, and contains multiple levels of data dependencies and parallelism. This paper presents an algorithm/architecture space exploration of efficient accelerators to achieve better network convergence rates and higher energy efficiency for training DNNs. We further demonstrate that an architecture with hierarchical support for collective communication semantics provides flexibility in training various networks performing both stochastic and batched gradient descent based techniques. Our results suggest that smaller networks favor non-batched techniques while performance for larger networks is higher using batched operations. At 45nm technology, CATERPILLAR achieves performance efficiencies of 177 GFLOPS/W at over 80% utilization for SGD training on small networks and 211 GFLOPS/W at over 90% utilization for pipelined SGD/CP training on larger networks using a total area of 103.2 mm$^2$ and 178.9 mm$^2$ respectively.

Introduction

While significant progress has been made in accelerating deep neural network (DNN) inference, the training phase remains a major computational bottleneck. Training is inherently more complex than inference, involving bidirectional data flow, intricate data dependencies, and a fundamental trade-off between algorithmic convergence and hardware efficiency.

Traditional training methods like Stochastic Gradient Descent (SGD) offer fast convergence per data pass but are often memory-bound due to their reliance on General Matrix-Vector multiplication (GEMV). Conversely, Minibatch Gradient Descent (MBGD) leverages General Matrix-Matrix multiplication (GEMM) for better hardware utilization but may require more epochs to reach target accuracy. CATERPILLAR addresses these challenges through a specialized Coarse Grain Reconfigurable Architecture (CGRA) designed to optimize both compute and communication for backpropagation.

Multilayer Perceptron (MLP) structure illustrating the layers and weights involved in the training process.

The CATERPILLAR Architecture

CATERPILLAR is built around an array of Linear Algebra Cores (LACs). Each LAC contains a 2D grid of Processing Elements (PEs) interconnected by high-speed row and column broadcast buses. These cores are further organized in a systolic ring to support efficient collective communication operations, such as all-gather and reduce-scatter, which are critical for synchronizing weights and gradients across the network.

Continuous Propagation (CP)

The core innovation of CATERPILLAR is "Continuous Propagation" (CP). Unlike standard training where one sample must complete a full forward and backward pass before the next begins, CP pipelines multiple samples through the network layers. This approach allows the hardware to maintain high utilization by processing different layers simultaneously.

Comparison of training strategies: (a) SGD, (b) MBGD, (c) Pipelined MBGD, and (d) Continuous Propagation (CP).

Furthermore, CP enhances weight locality. Updated weights can be applied to subsequent samples immediately after they are computed in the backward pass, reducing the frequency of weight fetches from external memory. This effectively mitigates the memory-bound nature of non-batched training methods.

Evaluation and Results

The architecture was evaluated using the MNIST dataset across various network configurations, ranging from 4 to 6 layers and up to 2500 neurons per layer. The results demonstrate that CATERPILLAR achieves exceptional energy efficiency and hardware utilization.

  • Performance: At 45nm technology, the system achieves 177 GFLOPS/W for SGD on smaller networks and up to 211 GFLOPS/W for pipelined training on larger networks.
  • Utilization: Hardware utilization exceeds 90% for large-scale networks, proving the effectiveness of the hierarchical communication and pipelining strategies.
  • Convergence: Empirical tests confirmed that CP maintains or improves upon the convergence rates of standard SGD, making it a highly effective algorithm for specialized hardware.

Conclusion

CATERPILLAR demonstrates that specialized hardware can bridge the gap between algorithmic efficiency and hardware performance in DNN training. By prioritizing communication and employing novel pipelining techniques like Continuous Propagation, it achieves high throughput and energy efficiency without requiring extreme memory bandwidth. Future work will explore extending this architecture to more complex network topologies, such as Convolutional and Recurrent Neural Networks, and investigating alternative training algorithms like Feedback Alignment.

Summary updated

Abstract

Accelerating the inference of a trained DNN is a well studied subject. In this paper we switch the focus to the training of DNNs. The training phase is compute intensive, demands complicated data communication, and contains multiple levels of data dependencies and parallelism. This paper presents an algorithm/architecture space exploration of efficient accelerators to achieve better network convergence rates and higher energy efficiency for training DNNs. We further demonstrate that an architecture with hierarchical support for collective communication semantics provides flexibility in training various networks performing both stochastic and batched gradient descent based techniques. Our results suggest that smaller networks favor non-batched techniques while performance for larger networks is higher using batched operations. At 45nm technology, CATERPILLAR achieves performance efficiencies of 177 GFLOPS/W at over 80% utilization for SGD training on small networks and 211 GFLOPS/W at over 90% utilization for pipelined SGD/CP training on larger networks using a total area of 103.2 mm$^2$ and 178.9 mm$^2$ respectively.
BibTeX
@misc{li2017caterpillar,
  author = {Yuan-Fang Li and Ardavan Pedram},
  title = {{CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks}},
  year = {2017},
  eprint = {1706.00517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.00517},
}
Figure