TinyMetaFed: Efficient Federated Meta-learning for TinyML

Haoyu Ren, Xue Li, Darko Anicic, Thomas A. Runkler

Abstract

TinyMetaFed is a novel framework designed to enable resource-constrained microcontrollers to perform federated meta-learning efficiently. By integrating online learning with selective communication strategies, TinyMetaFed allows low-power devices to collaboratively learn a robust model initialization that can be rapidly fine-tuned for diverse local tasks. This approach addresses the challenges of non-IID data and extreme resource scarcity, reducing energy consumption and communication overhead by over 50% compared to existing baselines.

Introduction

Tiny Machine Learning (TinyML) aims to deploy AI on low-power devices like microcontrollers, yet these environments suffer from a "data desert" and severe hardware limitations. Standard Federated Learning (FL) often fails here due to non-IID data distributions and high communication costs. Furthermore, traditional meta-learning algorithms, which "learn how to learn," are typically too computationally intensive for microcontrollers, requiring large data batches and complex higher-order derivatives. TinyMetaFed bridges this gap by providing a collaborative, adaptive learning framework tailored for the harsh realities of the Internet of Things (IoT).

Methods

TinyMetaFed is built upon Reptile, a first-order meta-learning algorithm that is significantly lighter than alternatives like MAML. The framework introduces several key innovations to optimize performance for TinyML:

  • Partial Local Reconstruction: The model is partitioned into global and local weights. Only global weights are synchronized with the server, while local weights remain on-device to capture task-specific nuances, enhancing privacy and reducing communication volume.
  • Top-P% Selective Communication: Instead of transmitting all updated weights, devices only send the "Top-P%"—the weights with the largest magnitude changes. This sparsification significantly reduces bandwidth usage.
  • Online Learning: TinyMetaFed processes data streams one sample at a time, eliminating the need to store large datasets in limited RAM.
  • Cosine Annealing: A learning rate schedule is employed to stabilize training and improve generalization to unseen tasks.
The TinyMetaFed workflow showing the interaction between the server and device.

Experimental Evaluation

The researchers evaluated TinyMetaFed across three domains using models from the MLPerf Tiny benchmark: a Sine-wave regression task, Omniglot image classification, and Keywords Spotting (KWS) audio classification. Testing was conducted on hardware including the Arduino Nano BLE 33 and Raspberry Pi 4.

Key Results

  • Energy and Communication Efficiency: TinyMetaFed achieved approximately 50% energy savings on image and audio tasks. Communication costs were reduced by 59% to 65% on the Omniglot and KWS datasets.
  • Convergence Performance: As demonstrated in the Sine-wave task, TinyMetaFed converges faster and maintains higher stability than baselines like TinyReptile.
Comparison of loss convergence for different algorithms on the Sine-wave task.

Discussion and Conclusion

TinyMetaFed demonstrates that meta-learning is a viable solution for heterogeneity in IoT by focusing on a "smart start" initialization rather than a single global model. While the framework provides inherent privacy benefits by keeping local weights on-device, future work will explore integrating Differential Privacy or Secure Aggregation to further harden the system. Additionally, further research into hyperparameter sensitivity will be essential for deploying TinyMetaFed in diverse industrial environments.

Summary updated

BibTeX
@inproceedings{ren2023tinymetafed,
  author = {Haoyu Ren and Xue Li and Darko Anicic and Thomas A. Runkler},
  title = {{TinyMetaFed: Efficient Federated Meta-learning for TinyML}},
  booktitle = {PKDD/ECML Workshops},
  year = {2023},
  doi = {10.1007/978-3-031-74640-6\_10},
}

Loading related content...

Figure