TDMiL: Tiny Distributed Machine Learning for Microcontroller-Based Interconnected Devices

Mayank Gulati, Koen Zandberg, Zhaolan Huang, Gerhard Wunder, Cédric Adjih, Emmanuel BaccelliIEEE Access 2024 · OpenReview

TL;DR

This paper introduces TDMiL, a novel framework for distributed machine learning on resource-constrained microcontrollers, enabling efficient training and transfer learning in IoT devices. It addresses the challenge of "stragglers" (slow devices) and offers a robust communication stack, paving the way for more intelligent and decentralized IoT systems.

What Problem Does This Paper Address?

The proliferation of Artificial Intelligence (AI) on edge devices, particularly in the Internet of Things (IoT), presents a significant challenge: how to perform complex machine learning tasks on devices with extremely limited computational power, memory, and battery life. Traditional machine learning relies on powerful servers, but for IoT, we need to decentralize this intelligence. While "TinyML" aims to shrink ML models for microcontrollers, a general framework for distributed training on these highly heterogeneous and resource-constrained devices has been missing. This paper tackles this gap by proposing TDMiL, a system designed to enable distributed machine learning on a diverse range of microcontrollers, specifically addressing the issue of varying device performance (stragglers) and the need for efficient communication protocols.

Methods

TDMiL introduces a novel communication stack built on CoAP (Constrained Application Protocol) and CBOR (Concise Binary Object Representation). CoAP is chosen for its reliability, asynchronous communication, and blockwise transfer capabilities, making it suitable for low-power, lossy IoT networks. CBOR is used for efficient data serialization, outperforming protobuf in some scenarios for arbitrary model sizes.

The framework also introduces "CoAPerator," a Python-based orchestrator that manages CoAP-enabled clients and interfaces with a model aggregator. This orchestrator facilitates registration, communication, and model distribution.

A key innovation is the handling of "stragglers"—devices that are significantly slower than others. TDMiL proposes two algorithms:

  • Algorithm 1 (Synchronous FedAvg): This approach discards updates from stragglers, reducing communication overhead but potentially losing valuable information.
  • Algorithm 2 (Asynchronous): This algorithm incorporates updates from stragglers, proportionally weighting their contributions, which is more robust for non-IID data but incurs higher communication costs.

The framework supports on-board training and transfer learning for various models, including simple MLPs and more complex architectures like LeNet5 and MobileNetV3-M.

Datasets

The paper uses several standard datasets for evaluation:

  • Iris Flowers: A small tabular dataset.
  • Wine: Another tabular dataset.
  • Breast Cancer: A tabular dataset.
  • MNIST: A widely used dataset for handwritten digit recognition.

For the tabular datasets, data augmentation (Gaussian noise) was applied due to their small size. The augmented datasets were split into IID (Independent and Identically Distributed) and Non-IID distributions across five clients to simulate real-world scenarios.

Key Insights and Ideas

  • CoAP and CBOR are a powerful combination for TinyML FL: Their lightweight nature and efficiency are crucial for overcoming the communication bottlenecks inherent in resource-constrained IoT networks.
  • Stragglers are a significant challenge, but manageable: While discarding straggler updates (Algorithm 1) reduces communication, incorporating them (Algorithm 2) is vital for stable convergence, especially in Non-IID scenarios.
  • Model size vs. performance trade-offs are critical: Even smaller models like LeNet5 can outperform larger, more complex models like MobileNetV3-M on certain tasks, highlighting the importance of model selection and optimization for microcontrollers.
  • The framework's generality is a strength: TDMiL is designed to be model-agnostic and compatible with various microcontrollers, making it a versatile tool for future research and development.

Key Results

  • High Accuracy Achieved: With IID data, both algorithms achieved over 96% Top-1 accuracy on the MNIST dataset.
  • Non-IID Performance Difference: Algorithm 2 (asynchronous) significantly outperformed Algorithm 1 (synchronous) in Non-IID scenarios, demonstrating stable convergence where Algorithm 1 struggled.
  • Communication Overhead: Algorithm 1 had lighter communication overhead due to discarding straggler updates.
  • Resource Consumption: The paper provides detailed measurements of storage, memory, and compute latency for different models on various microcontrollers, highlighting the significant impact of hardware architecture (e.g., the Raspberry Pi Pico's slower performance due to lack of FPU).
  • Model Size Impact: Transferring models like LeNet5 and MobileNetV3-M incurred substantial CoAP overhead, emphasizing the need for efficient serialization and communication.

Limitations and Future Work

  • Single Point of Failure: The current FL server (orchestrator) represents a single point of failure, limiting scalability and reliability. Future work aims to enable peer-to-peer aggregation without a central agent (e.g., Gossip Learning).
  • Privacy Concerns: While raw data is protected, model parameters themselves can be sensitive. The authors plan to incorporate privacy-enhancing techniques like Differential Privacy (DP) and secure aggregation using homomorphic encryption.
  • Hardware Agnosticism: While TDMiL clients are designed to be portable, the CoAPerator on Linux is not suitable for microcontrollers. Future work will focus on fully decentralized schemes for microcontroller-based devices.
  • Model Structure Updates: Updating the structure of deployed models is not possible without a firmware update, which is a limitation for dynamic model adaptation.

Summary updated

Abstract

More and more, edge devices embark Artificial Neuron Networks. In this context, a trend is to simultaneously decentralize their training as much as possible while shrinking their resource requirements, both for inference and training—tasks that are typically intensive in terms of data, memory, and computation. At the edge’s extremity, a specific challenge arises concerning the inclusion of microcontroller-based devices typically deployed in the IoT. So far, no general framework has been provided for that. Such devices not only have extremely challenging resource constraints (weak CPUs, slow network connections, memory budgets measured in kilobytes) but also exhibit high polymorphism, leading to large variability in computational performance among these devices. In this paper, we design and implement TDMiL, a versatile framework for distributed training, and transfer learning. TDMiL interconnects and combines logical components including CoAPerator (a central aggregator) and various tiny embedded software runtimes that are specifically tailored for networks comprising heterogeneous, resource-constrained devices built on diverse types of microcontrollers. We report on experiments conducted with the TDMiL framework, which we use to comparatively evaluate several schemes devised to address computational variability among distributed learning microcontroller-based devices, i.e., stragglers. Additionally, we release the code of our implementation of TDMiL as an open-source project, which is compatible with common commercial off-the-shelf IoT hardware and a well-known open-access IoT testbed.
BibTeX
@inproceedings{gulati2024tdmil,
  author = {Mayank Gulati and Koen Zandberg and Zhaolan Huang and Gerhard Wunder and Cédric Adjih and Emmanuel Baccelli},
  title = {{TDMiL: Tiny Distributed Machine Learning for Microcontroller-Based Interconnected Devices}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=TUjMAYZDLp},
}
Figure