Reduced precision floating-point optimization for Deep Neural Network On-Device Learning on microcontrollers

Davide Nadalini, Manuele Rusci, Luca Benini, Francesco Conti

Abstract

On-device learning (ODL) is essential for adapting pre-trained Deep Neural Networks (DNNs) to real-world data shifts in edge environments. However, the resource constraints of MicroController Units (MCUs)—specifically limited memory and low power budgets—often render standard training backpropagation impractical. This paper presents a high-performance software library designed for the Parallel Ultra Low Power (PULP) platform. By utilizing 16-bit floating-point (FP16) SIMD instructions and optimized parallel matrix kernels on a multi-core RISC-V architecture, the framework achieves training speeds up to 767× faster than existing single-core MCU solutions, enabling real-time local adaptation on battery-powered IoT devices.

Introduction

The standard paradigm for edge AI involves training models on powerful cloud-based GPUs and deploying them as static, "frozen" entities. While efficient, this approach fails when a device encounters data distributions different from its training set, such as a new user's voice or a unique physical environment. On-device learning (ODL) addresses this by allowing models to fine-tune locally.

Implementing ODL on MCUs is historically difficult. Most existing solutions are restricted to training only the final layer of a network or rely on 8-bit integer quantization, which can lead to training instability. This work introduces a full backpropagation implementation using reduced-precision floating point (FP16) to strike an optimal balance between numerical stability, memory footprint, and computational throughput.

Technical Methodology

The proposed library is optimized for the Greenwaves GAP9 SoC, which features an 8+1 core RISC-V cluster. The technical innovations focus on maximizing the utilization of the hardware's FP16 Single Instruction, Multiple Data (SIMD) capabilities.

FP16 SIMD and Parallelization

By transitioning from 32-bit to 16-bit floating point, the processor can execute two operations in a single cycle, effectively doubling the theoretical throughput. The framework distributes workloads across eight cores using aggressive loop unrolling to keep the Multiply-Accumulate (MAC) units saturated.

Optimized Matrix Operations

Standard matrix multiplication is a bottleneck in training due to the frequent need for data transposition during the backward pass. The authors developed specialized Transposed Matrix Multiplication ($MM_T$) kernels. By storing weights in a transposed format prior to deployment, the system avoids the "transposition tax" during runtime.

Furthermore, the library optimizes Im2Col and Im2Row transformations—essential for converting image data into matrices—to exploit the Height-Width-Channel (HWC) data layout. Unlike the traditional Channel-Height-Width (CHW) layout, HWC keeps channel data contiguous in memory, which is significantly more efficient for SIMD loading on RISC-V architectures.

Experimental Evaluation

The framework was evaluated using three representative TinyML benchmarks:

  1. ResNet8: For image classification tasks.
  2. DS-CNN (Depthwise Separable CNN): For keyword spotting in audio.
  3. MobileNet128: For a continual learning case study involving incremental class recognition.

Performance Results

The optimized FP16 kernels achieved a peak performance of 7.89 MAC/clk on the 8-core cluster, nearly doubling the efficiency of the FP32 baseline. In terms of end-to-end latency, a single training step for ResNet8 was completed in just 17.1 ms, while DS-CNN required only 6.4 ms.

When compared to the state-of-the-art, this approach is 1.6× faster than previous parallel FP32 implementations and over two orders of magnitude (767×) faster than the AIfES library running on a single-core MCU. Crucially, these tasks consumed an average of only 60.5 mW, confirming the feasibility of complex backpropagation on battery-powered nodes.

Discussion and Insights

One key finding is the "first-layer bottleneck." In layers with very few input channels (e.g., the initial 3-channel RGB input), the overhead of loop unrolling and SIMD setup can make FP16 slower than FP32. However, because these layers typically account for less than 3% of the total computational load, the overall system performance remains high.

Memory management remains a primary constraint. The 256KB L1 memory requires sophisticated tiling strategies to manage tensors between L1 and the larger L2 memory. Future work will explore mini-batching to improve training stability and investigate even lower precisions, such as FP8, to further reduce the memory and energy footprint.

Conclusion

This work demonstrates that high-speed, full-network training is no longer exclusive to high-power hardware. By optimizing for reduced-precision floating point and parallel RISC-V architectures, the PULP-TrainLib provides a foundation for truly autonomous, adaptive IoT systems. The project is available as open-source software to support the broader TinyML community.

Summary updated

BibTeX
@article{nadalini2023reduced,
  author = {Davide Nadalini and Manuele Rusci and Luca Benini and Francesco Conti},
  title = {{Reduced precision floating-point optimization for Deep Neural Network On-Device Learning on microcontrollers}},
  journal = {Future Generation Computer Systems},
  year = {2023},
  doi = {10.1016/j.future.2023.07.020},
}
Figure