Distributed machine learning across edge networks faces a fundamental tension between hardware constraints and training efficiency. Federated Learning (FL) allows thousands of clients to train models in parallel by aggregating local weight updates. However, FL hits a "memory wall" when models grow too large for resource-constrained edge devices like smart meters, IoT sensors, or smartphones.
To bypass this memory limitation, Split Learning (SL) cuts the neural network into sections. The client hosts only the first few layers (the client-side model), computes intermediate activations (known as "smashed data"), and transmits them to a powerful central server. The server holds the remaining layers, completes the forward pass, computes the loss, and backpropagates the gradients back to the client.
While this division of labor successfully offloads the computational burden, it introduces a severe sequential processing bottleneck. Traditional split learning architectures force a strict serialization of training, causing massive idle times and preventing distributed systems from scaling.
Anatomy of the Sequential Bottleneck
The sequential bottleneck in split learning manifests in two distinct dimensions: inter-client serialization and intra-round client-server dependency.
Traditional Split Learning (Sequential Client Execution):
Client 1: [ Forward ] -----------------------------> [ Backward ]
Server: [ Forward -> Loss -> Backward ]
Client 2: [ Forward ] ... (Waiting)
Intra-Round Dependency (Ping-Pong Idle Time):
Client: [ Compute Activations ] --------(Wait for Server)--------> [ Compute Gradients ]
Server: [ Compute Top Half of Network ]
1. Inter-Client Serialization (The Relay Baton Problem)
In standard split learning, clients must update the model one after another. Because the server maintains a single continuous state of the upper network layers, allowing multiple clients to send activations simultaneously without coordination would corrupt the gradient updates. As noted by Chandra Thapa et al. in their work on SplitFed: When Federated Learning Meets Split Learning, this relay-based approach means training time scales linearly with the number of participants. In large-scale networks, thousands of clients sit idle, waiting for their turn to interact with the server, creating prohibitive latency (Seungeun Oh et al.).
2. Intra-Round Serialization (The Ping-Pong Problem)
Even within a single client’s training iteration, the sequential nature of backpropagation creates a synchronization bottleneck. The client performs a forward pass, sends the smashed data, and must halt all computation while waiting for the server to execute its forward pass, calculate the loss, and return the gradients (Yiannis Papageorgiou et al.). This strict dependency prevents the overlapping of communication and computation, meaning network latency directly dictates wall-clock training speed (Yi Liu et al.).
Mechanism 1: Parallel Split Federated Learning (SFL)
To eliminate inter-client serialization, researchers developed Split Federated Learning (SFL). The core intuition is to combine the parallel execution paradigm of FL with the model-splitting benefits of SL.
Instead of processing clients sequentially, SFL allows all clients to compute their local forward passes concurrently. Each client sends its respective smashed data to the server simultaneously. The server processes these activations in parallel across distinct server-side model clones or threads.
Once the backward pass is completed for the batch, the client-side weights are aggregated and averaged using a federated mechanism (like FedAvg), while the server-side weights are similarly synchronized. Chandra Thapa et al. demonstrated that this hybrid framework drastically reduces training time bottlenecks while keeping the edge memory footprint minimal.
To further scale this parallel execution, Seungeun Oh et al. introduced Mix2SFL: Two-Way Mixup for Scalable, Accurate, and Communication-Efficient Split Federated Learning. Their framework uses a "two-way mixup" strategy: mixing intermediate activations during the forward pass and averaging gradients during the backward pass. This breaks the strict one-to-one client-server pairing bottleneck, allowing the system to handle massive distributed networks without a linear explosion in server compute requirements.
Mechanism 2: Decoupling the Backward Pass via Estimation
While SFL parallelizes multiple clients, it does not inherently solve the intra-round "ping-pong" idle time; clients still wait for the server's gradients to complete their local backward pass. To break this dependency, modern frameworks decouple the client and server training loops entirely using gradient estimation.
Decoupled Training with Auxiliary Models:
Client: [ Forward Pass ] -> [ Local Aux Loss ] -> [ Immediate Backward Pass ] -> (Next Batch)
|
(Smashed Data Sent)
v
Server: [ Server Forward & Backward Pass ]
In FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation, Srijith Nair et al. proposed utilizing local "auxiliary models" on the client side. These lightweight auxiliary layers sit on top of the client model during training to mimic the server's behavior and generate an immediate local loss. This allows the client to estimate the server's feedback and execute its backward pass instantly, moving on to the next training batch without waiting for the network round-trip.
A similar intuition drives the work of Cong Shen et al., who leverage auxiliary networks to execute local client updates independently, slashing server-side storage and communication delays by up to 90%. In a non-federated context, the authors of Hierarchical Training of Deep Neural Networks Using Early Exiting repurpose "early exit" branches within deep neural networks to provide immediate local gradients at the edge, completely isolating the client's backward pass from cloud dependencies.
Mechanism 3: Asynchronous and Adaptive Scheduling
In highly heterogeneous edge environments, a single slow client (a "straggler") or a dropped connection can stall an entire parallel training round. To make parallel split learning robust, researchers employ asynchronous architectures and dynamic resource allocation.
Decoupled Publisher/Subscriber Architectures
Instead of relying on synchronous handshaking and strict ID alignment between data pairs, Yi Liu et al. introduced PubSub-VFL. By utilizing a hierarchical, asynchronous publisher/subscriber queue, communication and computation are completely overlapped. The server processes incoming smashed data as a continuous stream, maximizing hardware utilization and preventing network latency from becoming the primary training bottleneck.
Dynamic Tiering and Adaptive Partitioning
Rather than splitting the network at a fixed layer for all devices, frameworks can dynamically adjust the workload based on real-time network conditions and client capacities:
- Dynamic Tiering: Seyed Mahmoud Sajjadi Mohammadabadi et al. developed a scheduling algorithm that allows slower edge devices to offload a larger portion of their computational layers to the server, minimizing the idle time of faster devices in the network.
- Adaptive Vehicular Partitioning: In fast-paced, high-density environments like vehicular edge networks, Lu Yu et al. and Xianke Qiang et al. utilize Deep Reinforcement Learning (DRL) to adaptively shift the model partition point between vehicles and roadside infrastructure, optimizing the trade-off between local energy consumption and wireless communication latency.
The Fundamental Trade-offs
Overcoming the sequential bottleneck is not a free lunch; it introduces distinct trade-offs between wall-clock speed, memory overhead, and convergence behavior.
Convergence vs. Latency in Non-IID Environments
Mathematically, sequential split learning behaves like a single continuous training run across data distributions. As proven by Yipeng Li et al., this sequential progression actually yields superior convergence guarantees compared to parallel averaging methods when client data is highly heterogeneous (non-IID).
However, this statistical advantage comes at the cost of extreme wall-clock latency. When moving to parallel split architectures to fix this latency, the system risks "catastrophic forgetting" or slower convergence due to client gradient conflict (Chao Huang).
Memory Efficiency vs. Communication Rounds
Another strategy to combat the memory wall is progressive or depth-wise block training, where a model is trained in sequential layer blocks rather than splitting the architecture between a client and a server (Yebo Wu et al., Kai Zhang et al.). While this eliminates client-server idle time and drastically reduces local RAM requirements, training layer-by-layer increases the total number of communication rounds required for full model convergence, trading cumulative network overhead for immediate execution speed.
Key Foundational Work
FedVMR: A New Federated Learning method for Video Moment Retrieval (2022) — contributed 2 concept elements to this direction.
LocFedMix-SL: Localize, Federate, and Mix for Improved Scalability, Convergence, and Latency in Split Learning (2022) — contributed 1 concept elements to this direction.
SplitFed: When Federated Learning Meets Split Learning (2022) — contributed 1 concept elements to this direction.