Deep Learning on FPGAs: Past, Present, and Future

Griffin Lacey, Graham W. Taylor, Shawki AreibiarXiv 2016 · OpenReview

TL;DR

This paper reviews the growing intersection of deep learning and Field-Programmable Gate Arrays (FPGAs) for hardware acceleration. It argues that FPGAs offer a compelling alternative to GPUs, particularly for their power efficiency and architectural flexibility, enabling novel optimizations and potentially lowering the barrier to entry for deep learning hardware development.

What Problem Does This Paper Address?

The paper addresses the critical need for efficient hardware acceleration in deep learning. As deep learning models become larger and data volumes explode, the computational demands for both training and inference are immense. While Graphics Processing Units (GPUs) have become the de facto standard for this acceleration, the authors propose Field-Programmable Gate Arrays (FPGAs) as a promising alternative. FPGAs offer unique advantages in terms of power efficiency and architectural customization, which could unlock new possibilities for deploying deep learning in resource-constrained environments or for exploring novel model architectures. The paper aims to bridge the gap between the deep learning community and FPGA development by highlighting current trends and future directions.

Methods

The paper provides a review and analysis of the current landscape of deep learning on FPGAs. It discusses the fundamental architectures of deep learning models, specifically Multi-layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs), and explains why their inherent parallelism makes them suitable for hardware acceleration. The authors then delve into the characteristics of FPGAs, contrasting them with General-Purpose Processors (GPPs) and Application-Specific Integrated Circuits (ASICs). A significant portion of the paper focuses on the evolution of FPGA design tools, particularly the adoption of High-Level Synthesis (HLS) and OpenCL, which aim to make FPGAs more accessible to software developers. The paper also reviews existing FPGA implementations of deep learning models, highlighting their design choices and performance.

Datasets

Not explicitly discussed in the paper, though the ImageNet 1K dataset is mentioned in the context of evaluating CNN performance.

Key Insights and Ideas

  • FPGA Flexibility for Novel Optimizations: Unlike fixed-architecture GPUs, FPGAs allow for custom hardware designs, enabling researchers to explore model-level optimizations that might not be feasible on GPUs. This could lead to more efficient and specialized deep learning accelerators.
  • Power Efficiency Advantage: FPGAs generally offer superior performance per watt compared to GPUs, making them ideal for large-scale server deployments and resource-limited embedded applications where power consumption is a critical factor.
  • Bridging the Gap with High-Level Tools: The increasing availability and maturity of High-Level Synthesis (HLS) tools and OpenCL support for FPGAs are democratizing FPGA development, making them more accessible to deep learning researchers and application scientists who are more familiar with software development practices.
  • Pipeline Parallelism Suitability: The feed-forward nature of deep learning models aligns well with the pipeline parallelism capabilities of FPGAs, allowing for highly efficient execution.
  • Trade-offs in FPGA Design: While FPGAs offer flexibility and power efficiency, they come with longer compile times (place and route) compared to software development, which can be a hurdle for rapid iteration during the research and development phase.

Key Results

The paper highlights several key findings and trends:

  • Performance Benchmarks: The authors cite Microsoft's work achieving a throughput of 134 images/second on ImageNet 1K using an FPGA, with an estimated increase to 233 images/second on newer hardware, all while consuming significantly less power (25W) compared to high-performing GPUs (235W).
  • Historical Context: The paper traces the evolution of FPGA-based neural network implementations from early, low-precision designs in the 1990s to more sophisticated, high-performance CNN accelerators today.
  • Tooling Advancements: The adoption of OpenCL by major FPGA vendors like Xilinx and Altera is a significant step towards making FPGAs more accessible for deep learning tasks.
  • Framework Support: Several popular deep learning frameworks (Caffe, Torch, Theano) are beginning to offer or have unofficial support for OpenCL, paving the way for easier integration with FPGAs.

Limitations and Future Work

  • Design Compile Times: The paper acknowledges that long compile times for FPGAs remain a significant challenge for researchers who need to iterate quickly on designs.
  • Design Size Constraints: While modern FPGAs are increasing in density, implementing very large neural networks can still be a challenge due to FPGA fabric limitations compared to ASICs.
  • Future Work: The authors suggest that FPGAs can effectively accommodate the growing trends in deep learning by providing architectural freedom for exploration and research. They emphasize the need for continued development in high-level abstraction tools and the integration of FPGAs into consumer and data center applications. The paper also points to the potential for FPGAs to offer superior alternatives for low-power compute clusters due to their reconfigurability and performance-per-watt advantages.

Summary updated

Abstract

The rapid growth of data size and accessibility in recent years has instigated a shift of philosophy in algorithm design for artificial intelligence. Instead of engineering algorithms by hand, the ability to learn composable systems automatically from massive amounts of data has led to ground-breaking performance in important domains such as computer vision, speech recognition, and natural language processing. The most popular class of techniques used in these domains is called deep learning, and is seeing significant attention from industry. However, these models require incredible amounts of data and compute power to train, and are limited by the need for better hardware acceleration to accommodate scaling beyond current data and model sizes. While the current solution has been to use clusters of graphics processing units (GPU) as general purpose processors (GPGPU), the use of field programmable gate arrays (FPGA) provide an interesting alternative. Current trends in design tools for FPGAs have made them more compatible with the high-level software practices typically practiced in the deep learning community, making FPGAs more accessible to those who build and deploy models. Since FPGA architectures are flexible, this could also allow researchers the ability to explore model-level optimizations beyond what is possible on fixed architectures such as GPUs. As well, FPGAs tend to provide high performance per watt of power consumption, which is of particular importance for application scientists interested in large scale server-based deployment or resource-limited embedded applications. This review takes a look at deep learning and FPGAs from a hardware acceleration perspective, identifying trends and innovations that make these technologies a natural fit, and motivates a discussion on how FPGAs may best serve the needs of the deep learning community moving forward.
BibTeX
@misc{lacey2016deep,
  author = {Griffin Lacey and Graham W. Taylor and Shawki Areibi},
  title = {{Deep Learning on FPGAs: Past, Present, and Future}},
  year = {2016},
  eprint = {1602.04283},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.04283},
}
Figure