ESFL: Efficient Split Federated Learning Over Resource-Constrained Heterogeneous Wireless Devices

Guangyu Zhu, Yiqin Deng, Xianhao Chen, Haixia Zhang, Yuguang Fang, Tan F. Wong

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across decentralized edge devices while preserving data privacy. However, the increasing complexity of modern deep neural networks (DNNs), such as VGG or GPT variants, often exceeds the computational and memory capacities of resource-constrained Internet of Things (IoT) devices.

Split Federated Learning (SFL) was proposed to mitigate this by partitioning the model: the edge device processes the initial layers, and a central server handles the remaining computation. Despite its potential, standard SFL typically employs a uniform "one-size-fits-all" split point for all devices. In heterogeneous wireless environments where devices possess varying hardware capabilities and network conditions, this rigid approach leads to the "straggler" problem, where the slowest device dictates the overall training latency.

The ESFL Framework

To address these challenges, the authors introduce Efficient Split Federated Learning (ESFL). This framework optimizes distributed training by dynamically tailoring the model split point and server resource allocation to the specific constraints of each device.

System architecture of Efficient Split Federated Learning (ESFL).

ESFL operates on three core pillars:

  1. Dynamic Model Splitting: Unlike static SFL, ESFL calculates a custom "cut layer" for every individual client. This decision is based on a real-time assessment of the device's local processing power and its current uplink/downlink bandwidth.
  2. Server-Side Resource Allocation: The central server manages "virtual servers" for each client. By intelligently allocating its own CPU/GPU cycles, the server ensures that offloaded computations are completed synchronously across the fleet.
  3. Iterative Optimization: The system formulates the training process as a Mixed-Integer Non-Linear Program (MINLP). To solve this NP-hard problem efficiently, the authors developed an Alternative Optimization algorithm that iteratively refines cut layers and resource allocations, typically reaching near-optimal solutions in fewer than 10 iterations.

Dynamic Model Splitting and Heterogeneity

The primary innovation of ESFL lies in its ability to harmonize a diverse network of devices. By assigning a smaller local workload to weaker devices (cutting the model earlier) and prioritizing them for server-side resources, ESFL ensures that all devices finish their training rounds simultaneously.

Visualization of heterogeneous model splitting for different users.

This approach effectively manages the privacy-efficiency trade-off. While raw data never leaves the device, the server's massive compute power is utilized more effectively than in standard FL or SL, without the communication bottlenecks often seen in naive split learning implementations.

Experimental Evaluation

The framework was rigorously tested using the CIFAR-10 dataset across three architectures: VGG13, VGG16, and VGG19. The researchers simulated four distinct resource scenarios, ranging from "Poor" to "Rich" compute and communication environments, to evaluate ESFL's robustness.

Key Results

  • Latency Reduction: In resource-constrained scenarios, ESFL reduced the one-round training time for VGG19 by approximately 41% compared to standard SFL and 50% compared to traditional FL.
  • Communication Efficiency: In scenarios with poor communication but rich server compute, ESFL achieved communication speeds nearly 10x faster than standard SFL. This was achieved by moving the cut layer to a point that minimized the size of the transmitted "smashed data" (activations).
  • Accuracy Preservation: Crucially, these efficiency gains do not compromise model performance. ESFL maintains the same convergence accuracy as standard Federated Learning.

Conclusion and Future Work

ESFL demonstrates that joint optimization of model partitioning and resource allocation can significantly accelerate distributed training in heterogeneous networks. While the current model assumes a stationary wireless environment and IID data distribution, it provides a robust foundation for future research. Potential extensions include integrating Non-Orthogonal Multiple Access (NOMA) protocols and testing the framework's resilience against highly non-IID (skewed) data distributions common in real-world edge deployments.

Summary updated

BibTeX
@article{zhu2024esfl,
  author = {Guangyu Zhu and Yiqin Deng and Xianhao Chen and Haixia Zhang and Yuguang Fang and Tan F. Wong},
  title = {{ESFL: Efficient Split Federated Learning Over Resource-Constrained Heterogeneous Wireless Devices}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2024.3397677},
}
Figure