RedMule: A mixed-precision matrix-matrix operation engine for flexible and energy-efficient on-chip linear algebra and TinyML training acceleration

Yvan Tortorella, Luca Bertaccini, Luca Benini, Davide Rossi, Francesco Conti

Introduction

While TinyML has successfully brought deep learning inference to low-power edge devices, the next frontier is on-device training. Enabling "near-sensor" training allows IoT devices to adapt to new data distributions or environmental changes locally, preserving privacy and reducing communication overhead. However, training presents a significantly higher computational hurdle than inference. Unlike inference, which often utilizes low-precision integer quantization (INT8), training requires the high dynamic range of floating-point (FP) arithmetic to accurately represent error gradients and weight updates.

Standard microcontrollers (MCUs) are generally ill-equipped for these tasks, often lacking robust floating-point units or the throughput necessary for matrix-heavy workloads. Furthermore, existing hardware accelerators are frequently designed with a rigid focus on standard neural network layers, lacking the flexibility to handle broader linear algebra problems. This paper introduces RedMule (Reduced-precision matrix Multiplication Engine), a specialized hardware accelerator designed to bridge the gap between high-performance training requirements and the stringent power constraints of the extreme edge.

Architecture and Methods

RedMule is integrated into a Parallel Ultra-Low-Power (PULP) cluster, operating alongside eight RISC-V cores. The engine is designed to maximize arithmetic intensity while maintaining the flexibility required for diverse mathematical kernels.

Generalized GEMM-Ops

Going beyond standard multiply-accumulate (MAC) operations, RedMule supports "Generalized" Matrix-Matrix Operations (GEMM-Ops). By allowing the replacement of the standard multiply/add pair with alternative operators such as min/max or add/max, the engine can accelerate non-AI algorithms, including shortest-path finding (graph analytics) and reliability analysis. This flexibility is achieved with a modest 16% area overhead compared to a standard matrix multiplier.

Mixed-Precision and Hybrid FP8

To balance accuracy and energy efficiency, RedMule utilizes a mixed-precision approach. While internal accumulation is performed in FP16 to maintain numerical stability, the engine features a casting module that supports Hybrid FP8 (HFP8) formats. It can process both E4M3 (optimized for activation precision) and E5M2 (optimized for gradient range) formats. This hybrid approach effectively doubles performance and halves memory bandwidth requirements compared to a pure FP16 implementation.

Outer-Product Microkernel

RedMule utilizes a 2D array of Computing Elements (CEs) implementing a block-dot (outer product) microkernel. This architectural choice is specifically intended to maximize the ratio of operations to memory accesses, ensuring that the engine remains compute-bound rather than memory-bound in typical edge scenarios.

Energy Management

To address the "leftover" problem—where matrix dimensions do not perfectly align with the hardware array size—RedMule employs hierarchical clock gating. This allows the system to dynamically disable unused rows and columns within the datapath, preventing unnecessary power consumption during the processing of non-square or small-dimension matrices.

Experimental Evaluation

The authors evaluated RedMule using a 22nm CMOS process across several benchmarks:

  • Synthetic GEMM: Testing peak throughput and efficiency across various matrix geometries.
  • ResNet8 (TinyMLPerf): A standard benchmark for edge devices used to evaluate training acceleration.
  • MobileNet V2: Demonstrating speedups in training depthwise and pointwise layers.
  • TinyTransformer: Evaluating FP8 inference to demonstrate the accelerator's versatility across modern architectures.

Key Results

RedMule demonstrates significant improvements in both raw performance and energy efficiency over software-only baselines:

  • Efficiency: At an operating point of 470 MHz (0.65 V), the cluster achieves 755 GFLOPS/W for FP16 and 920 GFLOPS/W for FP8. Specialized GEMM-Ops reach efficiencies as high as 1.67 TFLOPS/W.
  • Performance: The engine reaches a peak performance of 117 GFLOPS (FP8) while maintaining a power budget of less than 60 mW.
  • Acceleration: Compared to execution on eight RISC-V cores, RedMule provides a 15x speedup for standard large matrix multiplications and up to a 62x speedup for specialized GEMM-Ops. For a full ResNet8 training step, the combination of RedMule and a dedicated DataMover provides a 4.9x speedup over the software baseline.

Discussion and Limitations

While RedMule represents a significant step forward for TinyML training, several challenges remain for future exploration. As the Computing Element array scales (e.g., to 16x16 or 32x32), the memory interface requirements grow significantly, potentially necessitating 512-bit or wider buses, which complicates physical design. Additionally, RedMule is currently optimized for dense operations; hardware-level support for sparse matrix compression could further enhance efficiency for pruned models. Finally, the development of a robust compiler stack is necessary to automatically map high-level frameworks like PyTorch or TensorFlow to the specialized GEMM-Ops supported by the hardware.

In conclusion, RedMule proves that high-efficiency floating-point acceleration is viable at the milliwatt scale, providing the necessary computational foundation for the next generation of adaptive, self-learning IoT devices.

Summary updated

BibTeX
@article{tortorella2023redmule,
  author = {Yvan Tortorella and Luca Bertaccini and Luca Benini and Davide Rossi and Francesco Conti},
  title = {{RedMule: A mixed-precision matrix-matrix operation engine for flexible and energy-efficient on-chip linear algebra and TinyML training acceleration}},
  journal = {Future Generation Computer Systems},
  year = {2023},
  doi = {10.1016/j.future.2023.07.002},
}
Figure