Abstract
Results from the TinyML community demonstrate that it is possible to execute machine learning models directly on the terminals themselves, even if these are small microcontroller-based devices. However, to date, practitioners in the domain lack convenient all-in-one toolkits to help them evaluate the feasibility of executing arbitrary models on arbitrary low-power IoT hardware. To this effect, we present in this paper U-TOE, a universal toolkit we designed to facilitate the task of IoT designers and researchers, by combining functionalities from a low-power embedded OS, a generic model transpiler and compiler, an integrated performance measurement module, and an open-access remote IoT testbed. We provide an open source implementation of U-TOE and we demonstrate its use to experimentally evaluate the performance of various models, on a wide variety of low-power IoT boards, based on popular microcontroller architectures. U-TOE allows easily reproducible and customizable comparative evaluation experiments on a wide variety of IoT hardware all-at-once.
TL;DR
U-TOE (Universal TinyML On-board Evaluation) is an open-source toolkit designed to automate the deployment and performance profiling of machine learning models on low-power IoT hardware. By bridging the gap between high-level ML frameworks and bare-metal microcontrollers, it allows researchers to benchmark custom models across diverse hardware architectures—including ARM, RISC-V, and ESP32—at both the full-model and individual-operator levels.
The Challenge of TinyML Fragmentation
The "TinyML" revolution aims to run AI directly on the tiny microcontrollers (MCUs) found in sensors and wearables. However, the ecosystem is currently fragmented. If an engineer wants to know how a specific PyTorch model will perform on a specific ARM Cortex-M4 board versus a RISC-V board, they usually face a manual, uphill battle of cross-compilation, flashing, and custom instrumentation.
Existing benchmarking suites like MLPerf Tiny provide static results for fixed models, but they don't help developers evaluate their own custom architectures. Furthermore, most profiling tools are vendor-specific or limited to a single ML framework. U-TOE addresses this by providing a "universal" pipeline that handles everything from model transpilation to remote hardware execution and granular performance reporting.
Methodology: A Universal Pipeline
The authors built U-TOE by integrating several powerful open-source components into a unified workflow:
- Model Transpilation (uTVM): U-TOE leverages microTVM to convert high-level neural network models (from TensorFlow, PyTorch, etc.) into optimized C code or LLVM Intermediate Representation (IR) that can run on resource-constrained hardware.
- OS Environment (RIOT): Instead of running "bare-metal," U-TOE uses RIOT, a friendly operating system for the IoT. This provides a hardware abstraction layer, making the toolkit compatible with over 250 different types of boards out of the box.
- RPC Mechanism: To enable "Per-Operator" evaluation, U-TOE uses a Remote Procedure Call (RPC) system. This allows a host computer to trigger specific functions (like a single convolution layer) on the MCU over a serial connection and measure exactly how long that specific piece takes to execute.
- Remote Testbed Integration: U-TOE is designed to work with FIT IoT-LAB, a large-scale open-access testbed. This allows researchers to run experiments on hundreds of remote devices across the globe without needing physical hardware on their desk.
Experimental Evaluation
While the paper focuses on the toolkit rather than a new dataset, it validates U-TOE using a suite of representative TinyML models and tasks, including DS-CNN (Keyword Spotting), MobileNetV1 (Visual Wake Words), LeNet-5 (Image Classification), Deep AutoEncoder (Anomaly Detection), and RNNoise (Noise Suppression). These models were sourced from the MLPerf Tiny and ARM Model Zoo repositories to ensure industry-standard comparability.
The authors successfully demonstrated U-TOE's versatility across 14 different hardware platforms. For the LeNet-5 model, computational latency varied wildly based on the hardware—from a snappy 39.6 ms on the high-performance STM32F7 (216 MHz) to 262.1 ms on the ultra-low-power STM32L0 (32 MHz). The toolkit provides 95% confidence intervals for all measurements, ensuring that the performance data is statistically sound and reproducible.
Key Insights and Future Work
- The "Memory Wall": Testing on the SiFive RISC-V board revealed a massive performance regression when caches were disabled, highlighting that memory architecture is often more critical than raw CPU clock speed in TinyML.
- Architecture Optimization: ARM Cortex-M3 and M4 cores significantly outperformed M0+ cores at similar clock speeds due to specialized DSP instructions and the Thumb-2 instruction set.
- Granularity is Key: By profiling at the "operator" level, U-TOE revealed that single fused operators can account for over 80% of total execution time, allowing developers to target optimizations where they matter most.
Currently, U-TOE is optimized for single-core microcontrollers. As multi-core MCUs become more common, the toolkit will need to evolve to handle parallel execution profiling. Future versions may also integrate direct energy consumption measurement (mW) into the software stack, which is currently handled by external testbed hardware.
Summary updated