Distributed learning of deep neural network over multiple agents

Otkrist Gupta, Ramesh Raskar

Abstract: In sensitive domains like healthcare and finance, data privacy regulations often prevent the centralization of datasets required for training large-scale deep learning models. This paper presents "Split Learning," a distributed training technique that allows multiple agents to collaborate on model training without sharing raw data. By splitting the neural network architecture between a data owner (Alice) and a computing resource (Bob), the method minimizes local computational requirements and communication overhead compared to traditional Federated Learning. Experimental results on standard benchmarks demonstrate that Split Learning achieves performance parity with centralized training while offering significant efficiency gains.

1. Introduction

The proliferation of deep learning has created a massive demand for large, diverse datasets. However, in fields such as medical imaging or financial fraud detection, data is often siloed due to privacy laws (e.g., HIPAA, GDPR) and ethical constraints. Moving this data to a central server for training is frequently impossible. While Federated Learning (FL) has emerged as a solution by keeping data local, it often imposes heavy computational burdens on client devices and requires high communication bandwidth for model weight synchronization.

2. Problem Statement

Existing distributed learning paradigms face three primary challenges:

  • Centralization Risks: Sharing raw data exposes sensitive information to potential breaches and regulatory violations.
  • Computational Constraints: Many data owners (e.g., mobile devices or small clinics) lack the high-end GPUs/TPUs required to train state-of-the-art models locally.
  • Communication Bottlenecks: Sending full model parameters across networks in Federated Learning is inefficient, especially for massive architectures with millions of parameters.

3. Methodology: Split Learning

The proposed Split Learning framework addresses these issues by partitioning the neural network at a specific layer $n$.

3.1 The Split Architecture

The model is divided into two parts: a client-side portion (held by Alice) and a server-side portion (held by Bob). Alice processes her local data through the initial $n$ layers and transmits only the resulting activations to Bob. Bob then completes the forward pass through the remaining layers.

3.2 Training Dynamics

During the backward pass, Bob computes the loss and backpropagates gradients to the split layer. These gradients are sent back to Alice, who completes the backpropagation for her local layers. This process ensures that raw data never leaves Alice's premises.

Configurations for multi-agent split learning showing sequential weight transfer (snapshots) and the split forward/backward pass between Alice and Bob.

3.3 Multi-Agent Configurations

To scale to multiple data owners, the authors propose a round-robin approach. As illustrated in the figure above, Alice 1 trains the model for a set period and then passes her local weights (a "snapshot") to Alice 2. This transfer can occur directly via peer-to-peer communication or through a centralized coordinator. This sequential training allows the global model to learn from diverse datasets without any agent seeing another's raw data.

4. Experimental Evaluation

The researchers evaluated Split Learning using standard computer vision benchmarks, including MNIST, CIFAR-10, CIFAR-100, and ImageNet (ILSVRC 2012). They tested various architectures such as LeNet, VGG, and AlexNet to demonstrate the method's flexibility across different network topologies.

5. Key Results and Insights

  • Performance Parity: Split Learning achieved accuracy levels nearly identical to centralized training. For instance, on the MNIST dataset, the distributed approach reached 99.20% accuracy, compared to 99.18% for a single-machine baseline.
  • Resource Efficiency: Compared to Federated Averaging, Split Learning significantly reduced the computational load (FLOPs) on client devices. By offloading the majority of layers to Bob, Alice only needs to perform a fraction of the total computation.
  • Privacy through Configuration: The authors argue that the high dimensionality of hidden layers provides a "configuration shield." Reversing activations back to raw data is computationally infeasible due to the factorial number of possible neuron permutations in fully connected layers (e.g., $4096!$ combinations for a standard layer).

6. Conclusion and Future Work

Split Learning provides a scalable, privacy-preserving alternative to centralized and federated training. While the current sequential training model introduces a potential bottleneck for very large numbers of agents, it offers a practical path forward for data-sensitive industries. Future research will focus on rigorous information-theoretic security proofs and identity-hiding techniques to further harden the system against adversarial attacks.

Summary updated

BibTeX
@article{gupta2018distributed,
  author = {Otkrist Gupta and Ramesh Raskar},
  title = {{Distributed learning of deep neural network over multiple agents}},
  journal = {J. Netw. Comput. Appl.},
  year = {2018},
  doi = {10.1016/j.jnca.2018.05.003},
}
Figure