Introduction
Split Learning (SL) has emerged as a promising collaborative learning technique, particularly for resource-constrained IoT devices. By splitting a deep neural network into client-side and server-side portions, SL allows devices to perform feature extraction locally while offloading heavy computation to a server. However, unlike Federated Learning (FL), the SL research landscape has remained fragmented, lacking standardized benchmarking tools. This "Wild West" environment makes it difficult to compare algorithms fairly, especially under heterogeneous (Non-IID) data distributions where different clients possess very different data.
To address this, we introduce SLPerf, the first unified open-source research library designed to standardize SL benchmarking. SLPerf enables researchers to conduct reproducible experiments across various paradigms, datasets, and data distributions.
The SLPerf Architecture
SLPerf is built as a modular framework consisting of three primary layers, designed to abstract the complexities of distributed training while maintaining flexibility for researchers.
- High-level SL API: This layer allows users to configure experiments via simple files, selecting paradigms (Standalone vs. Distributed) and applications (Image Classification, NLP, Graph AI).
- Low-level SL API: The core engine of the system, managing server/client interactions, model definitions, loss calculations, and communication protocols.
- Tools: A suite for post-experiment analysis, providing visualization and structuralization of log files to track metrics like accuracy, communication costs, and latency.
Taxonomy of Split Learning Paradigms
The SLPerf framework categorizes the diverse SL landscape into three main groups:
- Model-Split-Only: Basic splitting strategies such as Vanilla SL, U-shape SL, and Parallel SL.
- Weight Aggregation-based: Paradigms that combine SL with FL-style model averaging, most notably SplitFed.
- Intermediate Data Aggregation-based: Methods that aggregate gradients or "smashed data" from multiple clients to improve training (e.g., SGLR, CutMixSL).
Benchmarking and Key Results
SLPerf was evaluated using diverse datasets including MNIST, CIFAR-10, UCI Adult (tabular), and Ogbg-molhiv (graph). Experiments were conducted in both IID and Non-IID settings using Dirichlet distributions to simulate data heterogeneity.
Performance under Data Heterogeneity
A critical finding is that most SL paradigms struggle significantly with Non-IID data. In extreme cases, Vanilla SL's accuracy on MNIST plummeted to 18.26% (compared to ~90% in IID settings). However, SplitFed emerged as the most robust "all-rounder," maintaining a high accuracy of 90.82% even in highly heterogeneous setups. In such scenarios, Vanilla SL can actually perform worse than simple local training on a device.
Communication and Convergence
The choice of "cut layer"—the point where the model is split—is the primary factor in communication efficiency. For instance, splitting after a pooling layer in LeNet significantly reduces the size of the "smashed data" transmitted. Furthermore, SplitFed demonstrated faster convergence than standard Federated Averaging (FedAvg) across various data partitions.
Conclusion
SLPerf provides a much-needed foundation for standardized research in Split Learning. By unifying disparate paradigms into a single framework, it reveals critical insights into the performance trade-offs of SL, particularly the robustness of weight-aggregation methods like SplitFed in Non-IID environments. Future work will focus on integrating more recent data aggregation methods, formalizing privacy-loss metrics, and developing specialized architectures for graph-based learning.
Summary updated