SplitFed Resilience to Packet Loss: Where to Split, that is the Question

Chamani Shiranthika, Zahra Hafezi Kafshgari, Parvaneh Saeedi, Ivan V. Bajic

TL;DR

This paper investigates how communication failures—specifically packet loss—impact Split Federated Learning (SFL), a hybrid AI training method. By testing different "split points" in a neural network, the researchers found that a deeper split (keeping more layers on the client side) significantly improves the model's resilience to data loss, ensuring more reliable performance in real-world, unstable network conditions.

What Problem Does This Paper Address?

Training large AI models on sensitive data (like medical records) is a balancing act between privacy and computing power. Federated Learning (FL) keeps data private by training locally on client devices, but it requires those devices to be powerful enough to run the entire model. Split Learning (SL) solves the power issue by splitting the model between a client and a server, but it's difficult to scale to many users.

Split Federated Learning (SFL) combines the two, but it introduces a new vulnerability: the communication link between the client and the server. In the real world, networks aren't perfect; data packets get lost. This paper asks a critical architectural question: "Where should we split the model to make it most resilient to these network errors?"

Methods

The authors used a U-Net architecture, a standard for medical image segmentation, and tested two different split strategies:

  1. Shallow Split: Only the very first and last layers reside on the client.
  2. Deep Split: More of the initial processing and final upsampling layers are kept on the client.
The Split U-Net architecture showing shallow vs. deep split points.

They simulated packet loss by "zeroing out" rows of data in the feature maps (forward pass) and gradient maps (backward pass) during training. They also tested five different parameter aggregation algorithms (like FedAvg and Fed-NCL) to see if the choice of algorithm could compensate for the loss.

The overall SFL system architecture, showing how models are split and aggregated.

Datasets

The study focused on a high-stakes medical task: human embryo image segmentation. They used a dataset of 781 images, requiring the model to identify five distinct biological components. This is a complex task where losing even small amounts of spatial data can lead to total failure in identifying delicate structures.

Key Insights and Ideas

  • The "Recovery" Advantage: The deep split outperformed the shallow split across the board. The authors suggest this is because the client-side "back-end" has more layers in a deep split, giving the model more capacity to "learn" how to recover or compensate for the missing data before it's finalized.
  • Skip Connections are Lifelines: In the deep split, the first "skip connection" (a shortcut that passes high-resolution features directly to the end of the network) was located entirely on the client. This meant these crucial features never had to cross the lossy network link, preserving vital information.
  • ReLU as a Natural Buffer: The researchers noted that models using ReLU activations are naturally somewhat robust to packet loss. Since ReLU already produces many zeros in its output, the "zeroing out" caused by packet loss is less of a shock to the system than it would be for other activation functions.
  • Packet Loss vs. Dropout: Interestingly, at low loss rates (up to 50%), the model remained surprisingly robust. The authors hypothesize that packet loss might act similarly to Dropout, a common regularization technique that intentionally ignores neurons to prevent overfitting.
Visual example of how 10% packet loss manifests as black horizontal lines in feature and gradient maps.

Key Results

The quantitative results were striking. While both split types performed similarly under perfect conditions (around 82-83% Mean Jaccard Index), their paths diverged sharply as network quality dropped:

  • Catastrophic Failure in Shallow Splits: At a 90% packet loss rate, the shallow split models essentially collapsed, with accuracy dropping toward zero.
  • Resilience in Deep Splits: Under the same 90% loss, the deep split models—while degraded—still maintained significantly higher accuracy, especially when only a few clients were experiencing the loss.
  • Statistical Significance: Using 125 pairwise t-tests, the authors confirmed that the deep split's advantage is statistically significant ($p < 0.05$) in every single scenario tested.
Graphs showing MJI (accuracy) vs. Packet Loss probability for various aggregation methods.

Limitations and Future Work

While the findings are robust for this specific setup, the authors acknowledge several areas for further exploration:

  • Simplified Loss Model: The study used "independent and identically distributed" (iid) loss. Real-world networks often suffer from "bursty" loss, where many packets are lost in a row.
  • Architecture Specificity: The results are tied to the U-Net architecture. Other types of networks (like Transformers) might behave differently.
  • Recovery Techniques: The study didn't use any advanced "packet loss concealment" (like interpolating missing data). Future work could combine smart split points with these data recovery algorithms to create even tougher decentralized systems.

Summary updated

Abstract

Decentralized machine learning has broadened its scope recently with the invention of Federated Learning (FL), Split Learning (SL), and their hybrids like Split Federated Learning (SplitFed or SFL). The goal of SFL is to reduce the computational power required by each client in FL and parallelize SL while maintaining privacy. This paper investigates the robustness of SFL against packet loss on communication links. The performance of various SFL aggregation strategies is examined by splitting the model at two points -- shallow split and deep split -- and testing whether the split point makes a statistically significant difference to the accuracy of the final model. Experiments are carried out on a segmentation model for human embryo images and indicate the statistically significant advantage of a deeper split point.
BibTeX
@inproceedings{shiranthika2023splitfed,
  author = {Chamani Shiranthika and Zahra Hafezi Kafshgari and Parvaneh Saeedi and Ivan V. Bajic},
  title = {{SplitFed Resilience to Packet Loss: Where to Split, that is the Question}},
  booktitle = {ISIC/Care-AI/MedAGI/DeCaF @ MICCAI},
  year = {2023},
  doi = {10.1007/978-3-031-47401-9\_35},
}

Loading related content...

Figure