TL;DR
The authors introduce CFU Playground, a full-stack, open-source framework for designing and evaluating machine learning accelerators on FPGAs for TinyML applications. By allowing developers to tightly couple Custom Function Units (CFUs) to a RISC-V soft CPU and iteratively co-design hardware and software, the framework achieves massive 55× to 75× speedups on resource-constrained devices.
What Problem Does This Paper Address?
Tiny Machine Learning (TinyML) aims to run ML models on ultra-low-power, heavily resource-constrained microcontrollers (MCUs) for always-on edge applications like keyword spotting or visual wake words. To meet strict latency and power budgets, specialized hardware acceleration is often required.
However, building custom ASICs requires massive upfront costs and time. While FPGAs offer a flexible alternative, existing hardware design workflows are often proprietary, fragmented, or force developers to design accelerators in isolation from the software stack. This isolation ignores system-level bottlenecks (the "AI Tax" of moving data around). The authors recognized a critical need for an agile, open-source framework that allows engineers to iteratively co-design and evaluate both the ML software and the hardware accelerator together on actual hardware.
Methods
CFU Playground provides a "deploy $\rightarrow$ profile $\rightarrow$ optimize" loop by stitching together an entirely open-source stack: TensorFlow Lite for Microcontrollers (TFLM), RISC-V GCC toolchains, LiteX/VexRiscv for SoC and CPU generation, and open-source FPGA synthesis tools (like Yosys).
The core technical approach revolves around the Custom Function Unit (CFU). Instead of building a large, discrete ML coprocessor, developers create a small piece of custom hardware logic that is tightly integrated directly into the RISC-V CPU's datapath. Developers can then write custom assembly instructions to trigger this hardware for specific ML bottlenecks (e.g., a specialized multiply-accumulate operation).
Additionally, the authors integrated Vizier, Google's open-source black-box optimization service, to automate Design Space Exploration (DSE). This allows the system to automatically test thousands of CPU and CFU configurations (tweaking cache sizes, multipliers, etc.) to find the optimal balance of FPGA resource usage and inference latency.
Datasets
The paper evaluates the framework using standard TinyML workloads:
- MobileNetV2 (MNV2): Used for Image Classification, quantized to 8-bit integers (int8).
- MLPerf Tiny Benchmark: Specifically, the Keyword Spotting (KWS) int8 model, which is an industry-standard benchmark for ultra-low-power ML.
Key Insights and Ideas
- The "AI Tax" can be bypassed: By tightly coupling the CFU directly into the CPU pipeline, the system avoids the heavy latency and memory overhead of offloading data to a discrete accelerator. This is a game-changer for latency-bound TinyML applications.
- A little custom hardware goes a long way: You don't need a massive, generalized Tensor Processing Unit. A tiny, model-specific custom instruction (like a specialized 1x1 convolution MAC) can yield exponential speedups while consuming minimal FPGA logic cells.
- Full-stack visibility is a superpower: By optimizing software (e.g., loop unrolling, informing the compiler of constants) and hardware (e.g., adding a 4-way SIMD MAC) simultaneously, developers can navigate severe memory and logic constraints that would block isolated hardware design.
- Richer design spaces yield better efficiency: Automated DSE proved that a smaller, stripped-down CPU paired with a highly specialized CFU often outperforms a larger, standard CPU, creating Pareto-optimal designs that wouldn't be obvious to a human engineer.
Key Results
- Image Classification (MobileNetV2): By targeting the
CONV_2D1x1 operations, the team achieved a 55× speedup, dropping the operation time from 5.5 seconds to just 0.10 seconds per inference on an Arty A7-35T FPGA. - Keyword Spotting (KWS): Deployed on a tiny Fomu FPGA (which is the size of a penny and fits inside a USB port), the team achieved a 75× overall speedup. Inference time plummeted from 2.5 minutes to under 2 seconds.
- Rapid Prototyping: The framework is highly agile. The 55× MobileNetV2 speedup was achieved by a single senior engineer working part-time over just five weeks, while the 75× KWS speedup was accomplished by an undergraduate intern in under four weeks.
Limitations and Future Work
- CPU Lock-in: Currently, the framework only supports the VexRiscv soft CPU, as it is the only one configured with the specific CFU interface used. The authors plan to expand support to other RISC-V cores.
- Memory Bottlenecks: The tightly coupled CFU currently relies on the CPU for data movement. Future work includes adding Direct Memory Access (DMA) directly to the CFU engine to further reduce latency.
- Automation: The authors suggest exploring High-Level Synthesis (HLS) pragmas to automatically identify execution hotspots in C code and automatically generate the corresponding CFUs and custom instruction calls.
- Energy Metrics: While the paper focuses heavily on latency and resource area, future work will explicitly study the optimization space for power and energy efficiency.
Summary updated