FPGAs-as-a-Service Toolkit (FaaST)

Dylan Sheldon Rankin, Jeffrey D. Krupa, Philip C. Harris, Maria Acosta Flechas, Burt Holzman, Thomas Klijnsma, Kevin Pedro, Nhan Tran, Scott Hauck, Shih-Chieh Hsu, Matthew Trahms, Kelvin Lin, Yu Lou, Ta-Wei Ho, Javier M. Duarte, Mia LiuH2RC@SC 2020 · DOI · OpenReview

TL;DR

This paper introduces FaaST, the first open-source toolkit for deploying FPGAs as a service, enabling high-performance computing for demanding fields like High Energy Physics (HEP). FaaST demonstrates that FPGAs can offer significant throughput improvements over GPUs for small, dense neural networks and comparable performance for larger ones, all while potentially offering better power efficiency.

What Problem Does This Paper Address?

The relentless growth in computing demands, particularly in fields like High Energy Physics (HEP), has outpaced the traditional improvements in CPU performance (Dennard scaling). This has driven the adoption of heterogeneous computing, where specialized hardware like GPUs and FPGAs act as accelerators. While GPU-accelerated computing "as a service" has seen significant development, FPGA-based solutions have lagged, largely due to the steep learning curve associated with hardware description languages (HDLs). This paper addresses the need for a more accessible and standardized way to integrate FPGAs into existing computing workflows, specifically by providing an "as a service" model that abstracts away the underlying hardware complexity.

Methods

The core of this work is the FPGAs-as-a-Service Toolkit (FaaST). FaaST leverages existing tools and frameworks to create a cohesive system for deploying FPGA-accelerated inference. Key components include:

  • High-Level Synthesis (HLS): Tools like hls4ml and Xilinx's Vitis Accel are used to translate high-level code (like C++) into FPGA hardware descriptions (HDL), lowering the barrier to entry for FPGA development.
  • Asynchronous Communication: The SONIC framework is employed for client-server communication, utilizing gRPC for asynchronous, non-blocking calls. This allows client applications to continue processing while waiting for FPGA results, maximizing resource utilization.
  • Server Design: The FaaST server architecture is designed to manage requests and offload computation to FPGAs. For smaller networks like FACILE, multiple FPGA "compute units" (CUs) are utilized on a single FPGA to improve parallelism. For larger networks like ResNet-50, the paper explores both Xilinx ML Suite and Azure Stack Edge (ASE) as server implementations.
  • Buffering and Scheduling: Techniques like optimizing DDR buffer sizes and managing execution queues are used to ensure efficient data transfer and kernel execution, especially for streaming data.

Datasets

The paper does not explicitly list datasets used for training the neural networks. However, it focuses on evaluating the performance of two specific neural network architectures:

  • FACILE: A small, fully-connected neural network used for regressing particle energy from time-sequence data. This serves as a benchmark for ultrafast, low-latency inference.
  • ResNet-50: A 50-layer residual network, specifically a version trained to classify jets originating from a top quark. This represents a larger, more complex network requiring longer latencies.

The evaluation uses these models to benchmark the FaaST system against existing GPU-based solutions and different FPGA hardware platforms.

Key Insights and Ideas

  • FPGAs as a Viable "As-a-Service" Option: The paper convincingly demonstrates that FPGAs can be effectively deployed as a service, offering a compelling alternative to GPUs, especially for specific workloads.
  • Order-of-Magnitude Speedups for Small Networks: For small, dense networks like FACILE, FPGAs can achieve throughput improvements exceeding an order of magnitude compared to GPUs as a service. This is attributed to FPGAs' inherent low-latency capabilities.
  • Competitive Performance for Large Networks: For larger convolutional networks like ResNet-50, FPGAs offer throughput comparable to GPUs, making them a strong contender even for more complex tasks.
  • Abstraction is Key: The FaaST toolkit, by abstracting FPGA complexity through HLS and standardized server interfaces, significantly lowers the barrier to entry for researchers and engineers.
  • CPU Bottlenecks are Real: Even with powerful FPGAs, the performance can be limited by the CPU's ability to handle gRPC requests and manage data transfers, highlighting the importance of efficient server-side software.
  • Scalability Challenges: While multiple FPGAs can be used, the benefits don't always scale linearly, potentially due to factors like the Python Global Interpreter Lock (GIL) in some server implementations.

Key Results

  • FACILE Performance: FaaST servers running FACILE achieved throughputs of up to 86 million inferences per second on an Alveo U250 and 36 million on an AWS EC2 F1 instance, significantly outperforming a T4 GPU (8 million inferences/sec).
  • ResNet-50 Performance: For ResNet-50, FaaST servers achieved throughputs of 1400 inferences/sec on AWS EC2 F1 (with 8 FPGAs) and 460 inferences/sec on an Azure Stack Edge. This was comparable to or better than a V100 GPU (1700 inferences/sec) and a T4 GPU (250 inferences/sec).
  • Scalability: A FaaST server running FACILE could serve approximately 1500 simultaneous clients on an AWS EC2 F1 instance before network bandwidth became a bottleneck. With a higher bandwidth Alveo U250, an estimated 3600 clients could be supported.
  • Latency: For ResNet-50 on Azure Stack Edge, average latencies ranged from 23.4 ms (single core) to 412.1 ms (200 cores), with networking effects causing occasional higher outliers.

Limitations and Future Work

  • CPU Bottleneck: The authors acknowledge that the gRPC server itself can become a bottleneck, limiting the overall throughput even when the FPGA is underutilized. They suggest using proxy servers and multiple threads to mitigate this.
  • Scalability of Multi-FPGA Systems: For ResNet-50, the performance did not scale proportionally with the number of FPGAs, potentially due to software limitations like the Python GIL. This suggests further investigation into optimized server implementations for multi-FPGA setups is needed.
  • Network Bandwidth: For FACILE, network bandwidth became a limiting factor at high client counts, indicating the need for high-bandwidth network interfaces for optimal performance.
  • Generalizability: While the toolkit is designed to be generic, the paper primarily focuses on ML applications. Future work could explore integrating other types of functional FPGA kernels into the FaaST framework.
  • Power Efficiency: Although mentioned as a potential advantage of FPGAs, the paper does not present quantitative power consumption comparisons between FPGA and GPU solutions. This would be a valuable area for future research.

Summary updated

Abstract

Computing needs for high energy physics are already intensive and are expected to increase drastically in the coming years. In this context, heterogeneous computing, specifically as-a-service computing, has the potential for significant gains over traditional computing models. Although previous studies and packages in the field of heterogeneous computing have focused on GPUs as accelerators, FPGAs are an extremely promising option as well. A series of workflows are developed to establish the performance capabilities of FPGAs as a service. Multiple different devices and a range of algorithms for use in high energy physics are studied. For a small, dense network, the throughput can be improved by an order of magnitude with respect to GPUs as a service. For large convolutional networks, the throughput is found to be comparable to GPUs as a service. This work represents the first open-source FPGAs-as-a-service toolkit.
BibTeX
@inproceedings{rankin2020fpgas,
  author = {Dylan Sheldon Rankin and Jeffrey D. Krupa and Philip C. Harris and Maria Acosta Flechas and Burt Holzman and Thomas Klijnsma and Kevin Pedro and Nhan Tran and Scott Hauck and Shih-Chieh Hsu and Matthew Trahms and Kelvin Lin and Yu Lou and Ta-Wei Ho and Javier M. Duarte and Mia Liu},
  title = {{FPGAs-as-a-Service Toolkit (FaaST)}},
  booktitle = {H2RC@SC},
  year = {2020},
  doi = {10.1109/h2rc51942.2020.00010},
}
Figure