Introduction
The rapid advancement of TinyML has demonstrated that machine learning models can be executed directly on microcontroller-based terminals. However, the ecosystem remains fragmented, creating a "feasibility gap" for practitioners. Bridging high-level ML frameworks (like TensorFlow or PyTorch) with a diverse array of low-power hardware (ARM, RISC-V, ESP32) often requires manual, vendor-specific efforts. Developers frequently lack the tools to determine if a model will fit within the strict RAM/Flash constraints of a specific device or how it will perform before committing to a hardware choice.
To address this, we present U-TOE (Universal TinyML On-Board Evaluation Toolkit), an open-source, all-in-one framework designed to automate the deployment and performance profiling of ML models across heterogeneous IoT hardware.
System Architecture and Methodology
U-TOE achieves universality by integrating two cornerstone open-source projects: uTVM (MicroTVM) for model transpilation and the RIOT operating system for a lightweight, multi-platform runtime environment supporting over 250 types of IoT boards.
The Build Pipeline
The toolkit automates the transition from a high-level model to on-device firmware. As shown in the build pipeline, uTVM converts the output of an ML framework into efficient C code or LLVM IR. This model library is then combined with the RIOT OS and the U-TOE module to compile the final firmware for the target device.
Evaluation Modes
A core feature of U-TOE is its Remote Procedure Call (RPC) mechanism, which facilitates communication between a host PC and the IoT device over a serial connection. This architecture supports two distinct levels of evaluation:
- Per-Model Evaluation: Measures end-to-end metrics such as total latency, RAM, and Flash consumption.
- Per-Operator Evaluation: Wraps individual layers (e.g., specific convolutions), allowing researchers to identify specific performance bottlenecks within a model.
Furthermore, U-TOE integrates with the FIT IoT-Lab, an open-access testbed, enabling large-scale experiments on hundreds of remote devices without requiring physical access to the hardware.
Key Insights and Experimental Results
The authors validated U-TOE by benchmarking standard TinyML models (including LeNet-5, MobileNetV1, and RNNoise) across 14 different IoT boards featuring ARM, RISC-V, and ESP32 architectures.
Hardware Sensitivity and the "Memory Wall"
Experiments revealed that TinyML performance is heavily influenced by memory access patterns. By disabling caches, the researchers highlighted how hardware-level storage bottlenecks—such as the use of external SPI NOR flash on certain RISC-V MCUs—can lead to significant performance regressions compared to ARM Cortex-M architectures with internal memory.
Granular Performance Analysis
The per-operator analysis provided critical visibility into model execution. In one instance, a single "add" layer in a TFLite model accounted for over 80% of total execution time. Such insights are vital for hardware-software co-design, helping engineers select the most cost-effective microcontroller for a specific task rather than over-provisioning.
Conclusion and Future Work
U-TOE provides a much-needed bridge between the ML and IoT communities, offering a reproducible and automated way to evaluate model feasibility. While currently focused on single-core inference, future development aims to support multi-core SoCs and on-device learning. By open-sourcing the toolkit, we hope to accelerate research toward fully exploiting the potential of edge computing.
Summary updated