CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAs

Shvetank Prakash, Tim Callahan, Joseph Bushagour, Colby R. Banbury, Alan V. Green, Pete Warden, Tim Ansell, Vijay Janapa Reddi

Introduction

Tiny Machine Learning (TinyML) represents a frontier in edge computing, aiming to execute complex machine learning models on ultra-low-power microcontrollers (MCUs). These "always-on" applications, such as keyword spotting or visual wake words, operate under extreme resource constraints. To meet the necessary latency and power budgets, developers often turn to specialized hardware acceleration.

However, traditional hardware design for TinyML faces significant hurdles. Custom ASIC development involves prohibitive costs and long lead times. While FPGAs offer a flexible middle ground, existing workflows are frequently proprietary or fragmented, forcing a separation between hardware and software design. This isolation often leads to the "AI Tax"—significant system-level bottlenecks caused by the overhead of moving data between a CPU and a discrete accelerator. The authors introduce CFU Playground to bridge this gap, providing an agile, open-source framework for the iterative co-design of ML software and hardware.

The CFU Playground Framework

CFU Playground provides a comprehensive "deploy → profile → optimize" loop by integrating a completely open-source stack. This includes TensorFlow Lite for Microcontrollers (TFLM) for the software runtime, RISC-V GCC toolchains, LiteX and VexRiscv for SoC and CPU generation, and open-source FPGA synthesis tools like Yosys.

Overview of the CFU Playground full-stack framework and its integration with Vizier for automated optimization.

A standout feature of the framework is its integration with Vizier, Google’s open-source black-box optimization service. Vizier automates Design Space Exploration (DSE), allowing the system to evaluate thousands of potential CPU and hardware configurations. By automatically adjusting parameters like cache sizes and multiplier types, the framework identifies Pareto-optimal designs that balance FPGA resource utilization against inference latency.

Technical Approach: Tightly Coupled CFUs

The core innovation of the framework is the Custom Function Unit (CFU). Rather than designing a large, independent ML coprocessor, developers create small, specialized hardware logic units that are integrated directly into the RISC-V CPU's datapath.

Diagram showing the tight coupling of the CFU to the CPU register file and ALU.

As shown in the architecture above, the CFU is connected to the CPU's register file. This allows developers to trigger hardware acceleration using custom assembly instructions for specific ML bottlenecks, such as specialized multiply-accumulate (MAC) operations. By keeping the accelerator within the CPU pipeline, the system avoids the latency and memory overhead typically associated with offloading data to external accelerators.

Evaluation and Key Results

The framework was evaluated using industry-standard TinyML workloads, including MobileNetV2 (MNV2) for image classification and the MLPerf Tiny Keyword Spotting (KWS) benchmark, both quantized to 8-bit integers (int8).

  • MobileNetV2: By targeting specific 1x1 convolution operations, the framework achieved a 55× speedup on an Arty A7-35T FPGA. Inference time for these operations dropped from 5.5 seconds to just 0.10 seconds.
  • Keyword Spotting: When deployed on the ultra-compact Fomu FPGA, the framework delivered a 75× overall speedup. Inference time was reduced from 2.5 minutes to under 2 seconds.

Beyond performance, the framework demonstrates remarkable agility. The 55× speedup for MobileNetV2 was achieved by a single engineer in five weeks, while an undergraduate intern achieved the 75× KWS speedup in less than a month.

Key Insights

  • Bypassing the AI Tax: Tightly coupling hardware to the CPU pipeline is essential for latency-bound TinyML applications, as it eliminates data movement overhead.
  • Efficiency through Specialization: Massive gains do not require large, generalized processors. Small, model-specific custom instructions can yield exponential speedups with minimal FPGA logic.
  • Full-Stack Visibility: Simultaneous optimization of software (e.g., loop unrolling) and hardware (e.g., SIMD MAC units) allows developers to overcome constraints that would be insurmountable in isolated design environments.

Future Work and Limitations

While highly effective, the current iteration of CFU Playground is primarily tied to the VexRiscv soft CPU. The authors plan to expand support to a wider variety of RISC-V cores. Additionally, while the CFU is currently dependent on the CPU for data movement, future versions may incorporate Direct Memory Access (DMA) to further reduce latency. Other areas for exploration include the use of High-Level Synthesis (HLS) to automate the generation of CFUs from C code and a more detailed study of power and energy efficiency metrics.

Summary updated

BibTeX
@inproceedings{prakash2023cfu,
  author = {Shvetank Prakash and Tim Callahan and Joseph Bushagour and Colby R. Banbury and Alan V. Green and Pete Warden and Tim Ansell and Vijay Janapa Reddi},
  title = {{CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAs}},
  booktitle = {ISPASS},
  year = {2023},
  doi = {10.1109/ispass57527.2023.00024},
}
Figure