U-TOE: Universal TinyML On-Board Evaluation Toolkit for Low-Power IoT

Zhaolan Huang, Koen Zandberg, Kaspar Schleiser, Emmanuel Baccelli

Introduction

The rapid advancement of TinyML has demonstrated that machine learning models can be executed directly on microcontroller-based terminals. However, the ecosystem remains fragmented, creating a "feasibility gap" for practitioners. Bridging high-level ML frameworks (like TensorFlow or PyTorch) with a diverse array of low-power hardware (ARM, RISC-V, ESP32) often requires manual, vendor-specific efforts. Developers frequently lack the tools to determine if a model will fit within the strict RAM/Flash constraints of a specific device or how it will perform before committing to a hardware choice.

To address this, we present U-TOE (Universal TinyML On-Board Evaluation Toolkit), an open-source, all-in-one framework designed to automate the deployment and performance profiling of ML models across heterogeneous IoT hardware.

System Architecture and Methodology

U-TOE achieves universality by integrating two cornerstone open-source projects: uTVM (MicroTVM) for model transpilation and the RIOT operating system for a lightweight, multi-platform runtime environment supporting over 250 types of IoT boards.

The Build Pipeline

The toolkit automates the transition from a high-level model to on-device firmware. As shown in the build pipeline, uTVM converts the output of an ML framework into efficient C code or LLVM IR. This model library is then combined with the RIOT OS and the U-TOE module to compile the final firmware for the target device.

The U-TOE build and deployment pipeline on the host machine.

Evaluation Modes

A core feature of U-TOE is its Remote Procedure Call (RPC) mechanism, which facilitates communication between a host PC and the IoT device over a serial connection. This architecture supports two distinct levels of evaluation:

  1. Per-Model Evaluation: Measures end-to-end metrics such as total latency, RAM, and Flash consumption.
  2. Per-Operator Evaluation: Wraps individual layers (e.g., specific convolutions), allowing researchers to identify specific performance bottlenecks within a model.
The RPC-based interaction between the host and device for per-model and per-operator profiling.

Furthermore, U-TOE integrates with the FIT IoT-Lab, an open-access testbed, enabling large-scale experiments on hundreds of remote devices without requiring physical access to the hardware.

Key Insights and Experimental Results

The authors validated U-TOE by benchmarking standard TinyML models (including LeNet-5, MobileNetV1, and RNNoise) across 14 different IoT boards featuring ARM, RISC-V, and ESP32 architectures.

Hardware Sensitivity and the "Memory Wall"

Experiments revealed that TinyML performance is heavily influenced by memory access patterns. By disabling caches, the researchers highlighted how hardware-level storage bottlenecks—such as the use of external SPI NOR flash on certain RISC-V MCUs—can lead to significant performance regressions compared to ARM Cortex-M architectures with internal memory.

Granular Performance Analysis

The per-operator analysis provided critical visibility into model execution. In one instance, a single "add" layer in a TFLite model accounted for over 80% of total execution time. Such insights are vital for hardware-software co-design, helping engineers select the most cost-effective microcontroller for a specific task rather than over-provisioning.

Conclusion and Future Work

U-TOE provides a much-needed bridge between the ML and IoT communities, offering a reproducible and automated way to evaluate model feasibility. While currently focused on single-core inference, future development aims to support multi-core SoCs and on-device learning. By open-sourcing the toolkit, we hope to accelerate research toward fully exploiting the potential of edge computing.

Summary updated

Abstract

Results from the TinyML community demonstrate that, it is possible to execute machine learning models directly on the terminals themselves, even if these are small microcontroller-based devices. However, to date, practitioners in the domain lack convenient all-in-one toolkits to help them evaluate the feasibility of executing arbitrary models on arbitrary low-power IoT hardware. To this effect, we present in this paper U-TOE, a universal toolkit we designed to facilitate the task of IoT designers and researchers, by combining functionalities from a low-power embedded OS, a generic model transpiler and compiler, an integrated performance measurement module, and an open-access remote IoT testbed. We provide an open source implementation of U-TOE and we demonstrate its use to experimentally evaluate the performance of various models, on a wide variety of low-power IoT boards, based on popular microcontroller architectures. U-TOE allows easily reproducible and customizable comparative evaluation experiments on a wide variety of IoT hardware all-at-once. The availability of a toolkit such as U-TOE is desirable to accelerate research combining Artificial Intelligence and IoT towards fully exploiting the potential of edge computing.
BibTeX
@inproceedings{huang2023u,
  author = {Zhaolan Huang and Koen Zandberg and Kaspar Schleiser and Emmanuel Baccelli},
  title = {{U-TOE: Universal TinyML On-Board Evaluation Toolkit for Low-Power IoT}},
  booktitle = {PEMWN},
  year = {2023},
  url = {https://openreview.net/forum?id=yeq5A5DVvf},
}
Figure