Keynotes

Kunle Olukotun, Kunle Bio2023 · DOI

This document is a collection of keynote abstracts from the 2023 33rd International Conference on Field-Programmable Logic and Applications (FPL). Rather than a single research paper, it presents a high-level vision from industry and academic leaders (Stanford, Intel, MIT, CERN, and AMD) on the future of reconfigurable computing.

TL;DR

The FPL 2023 keynotes outline a pivot toward Reconfigurable Dataflow Accelerators (RDAs) and Adaptive SoCs to meet the "petaFLOPS-scale" demands of foundation models and exascale scientific data. The central theme is bridging the gap between high-level algorithms and hardware-centric design to handle increasingly irregular, data-intensive workloads.

What Problem Does This Paper Address?

The collective abstracts address the "computational wall" hit by traditional architectures when facing three major shifts:

  1. The Foundation Model Era: Models like GPT-3 require hundreds of petaFLOPS for training and inference. As these models evolve, they are becoming more "irregular," featuring sparse data access and complex control flows that traditional GPUs struggle to handle efficiently.
  2. The Data Deluge: Projects like CERN’s High-Luminosity LHC (HL-LHC) will soon generate an exabyte of data annually, requiring a "green revolution" in computing to process data without an unsustainable carbon footprint.
  3. The Programmability Gap: There remains a significant barrier between algorithm-centric developers (who use Python/PyTorch) and hardware-centric FPGA design.

Methods

The keynotes propose several novel architectural and software approaches:

  • Reconfigurable Dataflow Accelerators (RDAs): Kunle Olukotun (Stanford) describes RDAs that exploit hierarchical dataflow parallelism. A new execution model called "Dataflow Threads" is introduced to specifically accelerate irregular applications.
  • Computing Near Storage: Arvind (MIT) proposes building low-power appliances using flash storage and hardware accelerators. By processing data directly in an SSD form factor, these appliances aim to be 10x cheaper than traditional server clusters for analyzing terabyte-size, sparse graphs.
  • Adaptive Architectures: Trevor Bauer (AMD) discusses the XDNA architecture, which integrates Xilinx’s FPGA and Adaptive SoC innovations into mainstream AMD portfolios to create a scalable platform for heterogeneous compute.
  • Advanced EDA Tools: Nabeel Shirazi (Intel) emphasizes the need for next-generation EDA (Electronic Design Automation) software to bridge the gap between high-level algorithms and silicon, citing tools like Vivado IP Integrator and Model Composer as precursors.

Datasets

While specific training sets are not the focus of these abstracts, the authors reference the massive scales at which their systems must operate:

  • Foundation Models: Massive unlabeled datasets used for training models with tens of billions of parameters.
  • High-Energy Physics: The HL-LHC detector data, estimated at one exabyte per year.
  • Graph Analytics: Terabyte-size, irregular, and sparse graphs used in social media and sensor data analysis.

Key Insights and Ideas

  • Sparsity is the Future: Future AI models will move away from dense computations toward irregular data access. RDAs are positioned as the solution because they minimize off-chip communication by keeping dataflow local and reconfigurable.
  • The "SSD Appliance" Concept: Instead of a 32-node server cluster, complex graph analysis could be performed on a specialized appliance that plugs into a laptop, drastically lowering the barrier to entry for big data analytics.
  • Hardware-Software Synergy: The integration of Xilinx into AMD and the focus on Intel’s Quartus software signal that the future of FPGAs is not as standalone chips, but as "adaptive" components deeply integrated into broader SoC ecosystems.
  • Sustainability as a Constraint: CERN’s Maria Girone highlights that exascale computing is no longer just about speed; it must be "green," focusing on carbon footprint reduction alongside performance.

Key Results

  • Cost Efficiency: "Near-storage" appliances are projected to be 10x cheaper than 16-32 node server clusters for specific data-intensive tasks.
  • Compute Requirements: The abstracts quantify the current state-of-the-art as requiring 100s of petaFLOPS for foundation model training.
  • Patent/Industry Impact: The speakers represent a massive footprint in the field, with Trevor Bauer (AMD) noting over 70 patents and Nabeel Shirazi (Intel) noting 43, underscoring the commercial maturity of these "adaptive" technologies.

Limitations and Future Work

  • The "Gap" Remains: Despite improvements in EDA tools, the transition from an algorithm-centric world to a hardware-centric world is still cited as a "major obstacle" for the next decade.
  • Complexity of Irregular Flow: While "Dataflow Threads" are proposed for irregular applications, the abstracts acknowledge that future foundation models will have even longer sequence lengths and more complex architectures, suggesting a moving target for hardware designers.
  • Scaling EDA: Intel’s keynote specifically identifies "EDA software" as a grand challenge, implying that current tools are not yet sufficient for the next generation of silicon design.

Summary updated

Abstract

Generative AI applications with their ability to produce natural language, computer code and images are transforming all aspects of society. These applications are powered by huge foundation models such as GTP-3 which are trained on massive unlabeled datasets. Foundation models have 10s of billions of parameters and have obtained state-of-the-art quality in natural language processing, vision and speech applications. These models are computationally challenging because they require 100s of petaFLOPS of computing capacity for training and inference. Future foundation models will have even greater capabilities provided by more complex model architectures with longer sequence lengths, irregular data access (sparsity) and irregular control flow. In this talk I will describe how the evolving characteristics of foundation models will impact the design of the optimized computing systems required for training and serving these models. In my talk, I will explain how Reconfigurable Dataflow Accelerators (RDAs) can accelerate a broad set of data intensive applications including foundation models. RDAs efficiently exploit the hierarchical dataflow parallelism that exists in ML models while minimizing off-chip communication. I will also explain how RDAs can be used to accelerate irregular applications using a new execution model called Dataflow Threads.
BibTeX
@misc{olukotun2023keynotes,
  author = {Kunle Olukotun and Kunle Bio and Olukotun},
  title = {{Keynotes}},
  year = {2023},
  doi = {10.1109/fpl60245.2023.00007},
}
Figure