Model CBOR Serialization for Federated Learning

Koen Zandberg, Mayank Gulati, Gerhard Wunder, Emmanuel BaccelliarXiv 2024 · OpenReview

TL;DR

This paper introduces TinyFL, a novel message framework for Federated Learning (FL) on resource-constrained devices like microcontrollers. By leveraging CBOR for message serialization, TinyFL significantly reduces message sizes (up to 75% smaller than JSON), making FL more feasible on devices with limited bandwidth and processing power.

What Problem Does This Paper Address?

Federated Learning (FL) enables training machine learning models across decentralized devices without sharing raw data, offering privacy benefits. However, current FL frameworks often rely on communication protocols and serialization formats (like JSON or gRPC with Protocol Buffers) that are too resource-intensive for small, low-power devices such as microcontrollers. These devices have limited memory, processing power, and often use low-throughput, lossy network links. This creates a significant barrier to deploying FL in many embedded systems and IoT applications. The paper addresses this gap by proposing a communication framework specifically designed for these constrained environments.

Methods

The core of the proposed solution is the TinyFL framework, which utilizes CBOR (Concise Binary Object Representation) for message serialization. CBOR is a binary data format designed to be small and efficient, making it well-suited for constrained environments. TinyFL aims to integrate seamlessly with existing FL workflows and leverage network protocol stacks already present in resource-constrained devices, such as CoAP (Constrained Application Protocol) and IEEE 802.15.4.

The framework defines specific message structures for FL operations, including:

  • Global Model Update: Sent from the server to clients.
  • Local Dataset Update: Sent from clients to the server, containing metadata about local training.
  • Local Model Update: Sent from clients to the server, containing the trained model parameters.

The paper also uses CDDL (Concise Data Definition Language) to describe these CBOR data structures, ensuring clarity and machine-readability.

Datasets

The paper does not train models on specific datasets. Instead, it evaluates the message sizes of the TinyFL framework using:

  • Idealized ML models: With varying numbers of parameters (4, 1000, and 10,000 floating-point numbers).
  • A real-world model: LeNet-5, a well-known model suitable for microcontrollers (approximately 45,000 parameters).

The evaluation compares CBOR-encoded messages against Protobuf and minified JSON to demonstrate the size reduction achieved by CBOR.

Key Insights and Ideas

  • CBOR is a game-changer for constrained FL: The binary nature of CBOR, combined with its efficient encoding of numerical values, leads to dramatically smaller message sizes compared to human-readable formats like JSON. This is crucial for devices with limited bandwidth.
  • Message frequency matters: The paper highlights that while large messages (like global model updates) are infrequent, smaller, more frequent messages (like local dataset updates) need to be exceptionally small to avoid overwhelming the network and draining battery power. TinyFL addresses this by keeping even these frequent messages compact.
  • Leveraging existing infrastructure: TinyFL's design emphasizes using existing network stacks (like CoAP) and firmware components, reducing the overhead of introducing new technologies and making integration easier.
  • Optimistic vs. Pessimistic sizing: The authors acknowledge that CBOR's size can vary based on the actual data values. They provide both optimistic (best-case) and pessimistic (worst-case) size estimates, offering a realistic view of performance.

Key Results

The primary quantitative result is the significant reduction in message sizes achieved by TinyFL using CBOR:

  • Up to 75% smaller messages compared to JSON for idealized models.
  • For the LeNet-5 model, CBOR-encoded messages were approximately 24% of the size of JSON-encoded messages.
  • Even for large models, the FL_Local_DataSet_Update messages remain small (8B to 28B with CBOR), fitting within a single frame on constrained networks.
  • The paper demonstrates that while large messages can exceed single-frame limits, they are infrequent, and strategies like CoAP blockwise transfer can handle them.

Limitations and Future Work

Limitations: * The paper focuses primarily on message serialization size and does not deeply analyze the computational overhead of CBOR encoding/decoding on microcontrollers, though it's implied to be manageable. * While the framework is presented, a full end-to-end implementation and deployment on actual hardware are not detailed.

Future Work: * Personalization: Exploring the possibility of personalizing global models on the client side before deployment for inference. * Adaptive Updates: Investigating methods to discard server updates and rely solely on local updates if global model performance deteriorates. * Resource Management: Further research into integrating these capabilities into constrained FL systems, considering compute resources, model complexity, and communication overhead. * Transferring Partial Models: Using the framework to transfer only specific layers or components of models for flexible and efficient learning scenarios.

Summary updated

Abstract

The typical federated learning workflow requires communication between a central server and a large set of clients synchronizing model parameters between each other. The current frameworks use communication protocols not suitable for resource-constrained devices and are either hard to deploy or require high-throughput links not available on these devices. In this paper, we present a generic message framework using CBOR for communication with existing federated learning frameworks optimised for use with resource-constrained devices and low power and lossy network links. We evaluate the resulting message sizes against JSON serialized messages where compare both with model parameters resulting in optimal and worst case serialization length, and with a real-world LeNet-5 model. Our benchmarks show that with our approach, messages are up to 75 % smaller in size when compared to the JSON alternative.
BibTeX
@misc{zandberg2024model,
  author = {Koen Zandberg and Mayank Gulati and Gerhard Wunder and Emmanuel Baccelli},
  title = {{Model CBOR Serialization for Federated Learning}},
  year = {2024},
  eprint = {2401.14056},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.14056},
}
Figure