Abstract
Distributed machine learning methods like Federated Learning (FL) and Split Learning (SL) enable training on private data without centralizing it. However, communication overhead remains a primary bottleneck. This paper provides a theoretical and quantitative comparison of communication efficiency between FL and SL. We demonstrate that while FL is efficient for small models with large local datasets, SL offers superior scalability as model complexity and client counts increase, following a characteristic 'Efficiency Hyperbola' boundary.
Introduction
Training deep neural networks usually requires centralized data, posing privacy risks for sensitive information like medical records or personal device usage. Distributed learning methods like FL and SL allow models to learn from data without it ever leaving the source. However, "moving the model instead of the data" comes with a high price: communication overhead. In a world of billion-parameter models and millions of edge devices, the bandwidth required to sync weights or activations can become a dealbreaker. This study addresses the lack of a clear framework for deciding which method is more efficient for a given real-world scenario.
Architectural Differences
The core difference between FL and SL lies in how the model is distributed and what information is communicated:
- Federated Learning (FL): Every client holds a full copy of the model. They train locally and send the entire set of weight updates to a central server, which averages them and sends the updated model back. Communication cost is proportional to the number of clients ($K$) and the total number of model parameters ($N$).
- Split Learning (SL): The model is physically split. The client holds the first few layers, and the server holds the rest. During training, the client sends only the output of its last layer (the "smashed layer") to the server. The server then sends back the gradients.
Efficiency Analysis: The Communication Trade-off
The authors mathematically model the total data transferred per training epoch for both methods, introducing a ratio ($\rho$) to determine the more efficient approach. If $\rho > 1$, Split Learning is more efficient; if $\rho < 1$, Federated Learning takes the lead.
Key Insights
- The Smashed Layer Advantage: In SL, the amount of data sent depends on the number of data points, not the size of the model. As models get "fatter" (more parameters), SL's efficiency advantage grows because it only cares about the activation tensor size at the split point.
- The Client Scaling Paradox: In FL, adding more clients linearly increases the total communication. In SL, adding clients doesn't necessarily increase the total data transferred per epoch because the total number of data samples ($p$) across the network might remain constant.
- The Efficiency Hyperbola: The authors identify a "rectangular hyperbola" boundary. On one side, the model is small enough that FL's weight-sharing is cheap. On the other, the model is so large that SL's activation-sharing is the only viable path.
Real-World Scenarios
The paper evaluates the methods across simulated parameter regimes representing three real-world use cases:
- Smartwatches: Millions of clients, varying model sizes. SL is generally superior here due to the high client count.
- Healthcare/Hospitals: Few clients, very large models. SL's advantage grows with model complexity.
- Biobanks: High number of clients, large models, and massive datasets. The choice depends heavily on the specific model-to-data ratio.
Conclusion and Future Work
While SL shows significant advantages for large-scale models and high client counts, FL remains the go-to for small models with massive local datasets. Future work should extend this analysis to include computational burdens on clients and the impact of network latency on the "chattiness" of SL versus the "bulkiness" of FL. Nevertheless, the identified efficiency boundary provides a robust guide for architecting distributed learning systems.
Summary updated