SLPerf: a Unified Framework for Benchmarking Split Learning

Tianchen Zhou, Zhanyi Hu, Bingzhe Wu, Cen ChenarXiv 2023 · DOI · arXiv

Introduction

Split Learning (SL) has emerged as a promising collaborative learning technique, particularly for resource-constrained IoT devices. By splitting a deep neural network into client-side and server-side portions, SL allows devices to perform feature extraction locally while offloading heavy computation to a server. However, unlike Federated Learning (FL), the SL research landscape has remained fragmented, lacking standardized benchmarking tools. This "Wild West" environment makes it difficult to compare algorithms fairly, especially under heterogeneous (Non-IID) data distributions where different clients possess very different data.

To address this, we introduce SLPerf, the first unified open-source research library designed to standardize SL benchmarking. SLPerf enables researchers to conduct reproducible experiments across various paradigms, datasets, and data distributions.

The SLPerf Architecture

SLPerf is built as a modular framework consisting of three primary layers, designed to abstract the complexities of distributed training while maintaining flexibility for researchers.

The modular architecture of SLPerf, illustrating the High-level API, Low-level API, and analysis tools.
  1. High-level SL API: This layer allows users to configure experiments via simple files, selecting paradigms (Standalone vs. Distributed) and applications (Image Classification, NLP, Graph AI).
  2. Low-level SL API: The core engine of the system, managing server/client interactions, model definitions, loss calculations, and communication protocols.
  3. Tools: A suite for post-experiment analysis, providing visualization and structuralization of log files to track metrics like accuracy, communication costs, and latency.

Taxonomy of Split Learning Paradigms

The SLPerf framework categorizes the diverse SL landscape into three main groups:

  • Model-Split-Only: Basic splitting strategies such as Vanilla SL, U-shape SL, and Parallel SL.
  • Weight Aggregation-based: Paradigms that combine SL with FL-style model averaging, most notably SplitFed.
  • Intermediate Data Aggregation-based: Methods that aggregate gradients or "smashed data" from multiple clients to improve training (e.g., SGLR, CutMixSL).
Visual representation of (a) U-shaped SL and (b) SplitFed paradigms.

Benchmarking and Key Results

SLPerf was evaluated using diverse datasets including MNIST, CIFAR-10, UCI Adult (tabular), and Ogbg-molhiv (graph). Experiments were conducted in both IID and Non-IID settings using Dirichlet distributions to simulate data heterogeneity.

Performance under Data Heterogeneity

A critical finding is that most SL paradigms struggle significantly with Non-IID data. In extreme cases, Vanilla SL's accuracy on MNIST plummeted to 18.26% (compared to ~90% in IID settings). However, SplitFed emerged as the most robust "all-rounder," maintaining a high accuracy of 90.82% even in highly heterogeneous setups. In such scenarios, Vanilla SL can actually perform worse than simple local training on a device.

Communication and Convergence

The choice of "cut layer"—the point where the model is split—is the primary factor in communication efficiency. For instance, splitting after a pooling layer in LeNet significantly reduces the size of the "smashed data" transmitted. Furthermore, SplitFed demonstrated faster convergence than standard Federated Averaging (FedAvg) across various data partitions.

Conclusion

SLPerf provides a much-needed foundation for standardized research in Split Learning. By unifying disparate paradigms into a single framework, it reveals critical insights into the performance trade-offs of SL, particularly the robustness of weight-aggregation methods like SplitFed in Non-IID environments. Future work will focus on integrating more recent data aggregation methods, formalizing privacy-loss metrics, and developing specialized architectures for graph-based learning.

Summary updated

Abstract

Data privacy concerns has made centralized training of data, which is scattered across silos, infeasible, leading to the need for collaborative learning frameworks. To address that, two prominent frameworks emerged, i.e., federated learning (FL) and split learning (SL). While FL has established various benchmark frameworks and research libraries,SL currently lacks a unified library despite its diversity in terms of label sharing, model aggregation, and cut layer choice. This lack of standardization makes comparing SL paradigms difficult. To address this, we propose SLPerf, a unified research framework and open research library for SL, and conduct extensive experiments on four widely-used datasets under both IID and Non-IID data settings. Our contributions include a comprehensive survey of recently proposed SL paradigms, a detailed benchmark comparison of different SL paradigms in different situations, and rich engineering take-away messages and research insights for improving SL paradigms. SLPerf can facilitate SL algorithm development and fair performance comparisons. The code is available at https://github.com/Rainysponge/Split-learning-Attacks .
BibTeX
@misc{zhou2023slperf,
  author = {Tianchen Zhou and Zhanyi Hu and Bingzhe Wu and Cen Chen},
  title = {{SLPerf: a Unified Framework for Benchmarking Split Learning}},
  year = {2023},
  eprint = {2304.01502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.01502},
}
Figure