Leveraging Automated Mixed-Low-Precision Quantization for Tiny Edge Microcontrollers

Manuele Rusci, Marco Fariselli, Alessandro Capotondi, Luca Benini

This paper presents a specialized framework designed to squeeze high-performance Deep Neural Networks (DNNs) onto the most resource-constrained hardware: MicroController Units (MCUs). By developing an automated Reinforcement Learning (RL) framework, the authors find optimal "mixed-precision" (2, 4, or 8-bit) quantization for each layer of a neural network, enabling state-of-the-art performance on hardware with extremely limited memory.

The Challenge of TinyML

Deep Learning at the "edge" typically targets mobile phones or specialized AI accelerators. However, the "TinyML" frontier involves MCUs—low-power chips found in thermostats, wearables, and industrial sensors. These devices face two primary hurdles:

  1. The Memory Wall: MCUs typically have less than 2MB of FLASH for model weights and 1MB of RAM for intermediate activations. Standard 8-bit quantization is often insufficient to fit modern models like MobileNet.
  2. Arithmetic Constraints: Most MCUs lack floating-point units and rely on integer-only arithmetic. Existing automated tools often use non-linear or clustering-based methods that are computationally impossible for simple MCUs to execute efficiently.

The authors build upon the Hardware-Aware Quantization (HAQ) framework, modifying it for the architectural realities of MCUs. The core of their approach is an RL-driven search stage followed by fine-tuning.

The automated search stage uses an RL agent to explore quantization policies, followed by a fine-tuning phase to recover accuracy.

Key Methodological Components:

  • RL-Driven Search: A Reinforcement Learning agent explores the massive search space of quantization policies, deciding the bit-width for weights and activations for every layer.
  • Uniform Quantization: Unlike previous works using K-Means clustering, this framework enforces Uniform Linear Quantization. This ensures the resulting model can run using standard integer-only libraries like CMSIS-NN (for ARM) or PULP-NN (for RISC-V).
  • Hardware Constraints: The search is restricted to power-of-two bit-widths (2, 4, and 8-bit) to maintain hardware efficiency. The RL agent is penalized if a policy exceeds specific FLASH ($M_{ROM}$) or RAM ($M_{RAM}$) budgets.
  • Proxy Datasets: To make the RL search computationally feasible, the authors used a proxy version of the ImageNet (ILSVRC 2012) dataset, consisting of 20,000 random samples for training and 10,000 for validation during the search phase.

Key Insights and Results

A major finding of the study is that for MCU-targeted models, restricted uniform quantization (2/4/8-bit) performs as well as, or better than, complex non-uniform methods. This validates the use of simpler integer-only arithmetic without sacrificing accuracy.

Comparison of different quantization schemes across MobileNet and MNasNet architectures.

Performance Breakthroughs:

  • MobileNetV1 Success: Under a strict constraint of 2MB FLASH and 512kB RAM, the framework produced a mixed-precision MobileNetV1 that achieved 68.4% Top-1 accuracy. This is 4% more accurate than a standard 8-bit uniform model (MNasNet 0.35) fitting the same memory footprint.
  • Compression Ratios: The framework achieved up to 9x compression for weights and 5.1x compression for activations compared to the 8-bit baseline.
  • Layer Sensitivity: In models with residual connections (MobileNetV2, MNasNet), the authors found that quantizing residual branches too aggressively destroys accuracy. The RL agent naturally learns to keep these sensitive layers at 8-bit while compressing others to "fill" the available memory budget.

Limitations and Future Outlook

While weight compression was highly successful, quantizing activations in models with residual connections led to significant accuracy drops (up to 17%). The authors suggest that better handling of parallel branches in the memory model is required for these complex architectures. Additionally, future work could incorporate inference latency directly into the RL reward function, as 2-bit or 4-bit operations may introduce software overhead on certain CPUs despite their memory benefits.

Summary updated

Abstract

The severe on-chip memory limitations are currently preventing the deployment of the most accurate Deep Neural Network (DNN) models on tiny MicroController Units (MCUs), even if leveraging an effective 8-bit quantization scheme. To tackle this issue, in this paper we present an automated mixed-precision quantization flow based on the HAQ framework but tailored for the memory and computational characteristics of MCU devices. Specifically, a Reinforcement Learning agent searches for the best uniform quantization levels, among 2, 4, 8 bits, of individual weight and activation tensors, under the tight constraints on RAM and FLASH embedded memory sizes. We conduct an experimental analysis on MobileNetV1, MobileNetV2 and MNasNet models for Imagenet classification. Concerning the quantization policy search, the RL agent selects quantization policies that maximize the memory utilization. Given an MCU-class memory bound of 2MB for weight-only quantization, the compressed models produced by the mixed-precision engine result as accurate as the state-of-the-art solutions quantized with a non-uniform function, which is not tailored for CPUs featuring integer-only arithmetic. This denotes the viability of uniform quantization, required for MCU deployments, for deep weights compression. When also limiting the activation memory budget to 512kB, the best MobileNetV1 model scores up to 68.4% on Imagenet thanks to the found quantization policy, resulting to be 4% more accurate than the other 8-bit networks fitting the same memory constraints.
BibTeX
@inproceedings{rusci2020leveraging,
  author = {Manuele Rusci and Marco Fariselli and Alessandro Capotondi and Luca Benini},
  title = {{Leveraging Automated Mixed-Low-Precision Quantization for Tiny Edge Microcontrollers}},
  booktitle = {IoT Streams/ITEM@PKDD/ECML},
  year = {2020},
  doi = {10.1007/978-3-030-66770-2\_22},
}
Figure