Detailed comparison of communication efficiency of split learning and federated learning

Abhishek Singh, Praneeth Vepakomma, Otkrist Gupta, Ramesh RaskararXiv 2019 · arXiv

Abstract

Distributed machine learning methods like Federated Learning (FL) and Split Learning (SL) enable training on private data without centralizing it. However, communication overhead remains a primary bottleneck. This paper provides a theoretical and quantitative comparison of communication efficiency between FL and SL. We demonstrate that while FL is efficient for small models with large local datasets, SL offers superior scalability as model complexity and client counts increase, following a characteristic 'Efficiency Hyperbola' boundary.

Introduction

Training deep neural networks usually requires centralized data, posing privacy risks for sensitive information like medical records or personal device usage. Distributed learning methods like FL and SL allow models to learn from data without it ever leaving the source. However, "moving the model instead of the data" comes with a high price: communication overhead. In a world of billion-parameter models and millions of edge devices, the bandwidth required to sync weights or activations can become a dealbreaker. This study addresses the lack of a clear framework for deciding which method is more efficient for a given real-world scenario.

Architectural Differences

The core difference between FL and SL lies in how the model is distributed and what information is communicated:

  • Federated Learning (FL): Every client holds a full copy of the model. They train locally and send the entire set of weight updates to a central server, which averages them and sends the updated model back. Communication cost is proportional to the number of clients ($K$) and the total number of model parameters ($N$).
  • Split Learning (SL): The model is physically split. The client holds the first few layers, and the server holds the rest. During training, the client sends only the output of its last layer (the "smashed layer") to the server. The server then sends back the gradients.
Architecture of Split Learning showing the division between client and server at the 'smashed layer'.

Efficiency Analysis: The Communication Trade-off

The authors mathematically model the total data transferred per training epoch for both methods, introducing a ratio ($\rho$) to determine the more efficient approach. If $\rho > 1$, Split Learning is more efficient; if $\rho < 1$, Federated Learning takes the lead.

The efficiency boundary between Federated Learning and Split Learning as a function of model parameters and number of clients.

Key Insights

  1. The Smashed Layer Advantage: In SL, the amount of data sent depends on the number of data points, not the size of the model. As models get "fatter" (more parameters), SL's efficiency advantage grows because it only cares about the activation tensor size at the split point.
  2. The Client Scaling Paradox: In FL, adding more clients linearly increases the total communication. In SL, adding clients doesn't necessarily increase the total data transferred per epoch because the total number of data samples ($p$) across the network might remain constant.
  3. The Efficiency Hyperbola: The authors identify a "rectangular hyperbola" boundary. On one side, the model is small enough that FL's weight-sharing is cheap. On the other, the model is so large that SL's activation-sharing is the only viable path.

Real-World Scenarios

The paper evaluates the methods across simulated parameter regimes representing three real-world use cases:

  • Smartwatches: Millions of clients, varying model sizes. SL is generally superior here due to the high client count.
  • Healthcare/Hospitals: Few clients, very large models. SL's advantage grows with model complexity.
  • Biobanks: High number of clients, large models, and massive datasets. The choice depends heavily on the specific model-to-data ratio.

Conclusion and Future Work

While SL shows significant advantages for large-scale models and high client counts, FL remains the go-to for small models with massive local datasets. Future work should extend this analysis to include computational burdens on clients and the impact of network latency on the "chattiness" of SL versus the "bulkiness" of FL. Nevertheless, the identified efficiency boundary provides a robust guide for architecting distributed learning systems.

Summary updated

Abstract

We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.
BibTeX
@misc{singh2019detailed,
  author = {Abhishek Singh and Praneeth Vepakomma and Otkrist Gupta and Ramesh Raskar},
  title = {{Detailed comparison of communication efficiency of split learning and federated learning}},
  year = {2019},
  eprint = {1909.09145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.09145},
}
Figure