PSL-RGHA: Parallel Split Learning with Reference Gradient Half-Space Alignment for Mitigating Gradient Divergence

Han Li, Zheng Lin, Miao Yang, Zehang Lin, Jing Yang, Zihan Fang, Qiyang Zhang, Zhaolu Kang, Jianmin Li, Lin Li, Mingwei Hong, Shunzhi ZhuarXiv 2026 · arXiv

TL;DR

PSL-RGHA is a novel framework that stabilizes parallel split learning in heterogeneous data environments by aligning client gradients on the server side. By combining robust reference gradient construction with geometric half-space projection, it eliminates training divergence and significantly boosts accuracy without requiring expensive client-side model synchronization.

What Problem Does This Paper Address?

Deploying deep learning on resource-constrained IoT devices is a major challenge. Standard Federated Learning (FL) requires devices to train full models, which often exceeds their memory and compute limits. Split Learning (SL) offers a solution by partitioning the model: the client handles the initial layers, and the server handles the rest.

The most efficient version, Parallel Split Learning (PSL), allows multiple clients to train in parallel without the bottleneck of periodic client-side parameter synchronization. However, this efficiency comes at a cost: training instability. When data is "non-IID" (heterogeneous across devices), local models quickly overfit to their specific data. Without synchronization, they send "semantically inconsistent" activations to the server. This results in wildly divergent and conflicting gradients during backpropagation, leading to severe client drift or total convergence failure.

Figure 1 shows how PSL converges slower and less stably than Split Federated Learning (SFL) under non-IID settings.

Methods

The authors propose PSL-RGHA, a framework that fixes these conflicts entirely on the server side using two synergistic modules:

  1. Reference Gradient Progressive Construction (RGPC): Instead of simply averaging client gradients (which is sensitive to outliers), RGPC first ranks them based on a "directional consistency score." It then progressively synthesizes a robust reference gradient using Weiszfeld updates—a method for finding a geometric median that is resilient to noise. This reference gradient acts as a "trusted" proxy for the true global descent direction.
  2. Gradient Half-Space Alignment (GHA): Once the reference direction is established, GHA examines each client's gradient. If a gradient conflicts with the global trend (i.e., it points in an opposing direction), GHA projects it onto the non-negative half-space defined by the reference. This effectively "neutralizes" components that would impede convergence while preserving the unique, constructive information from each client.
An overview of the PSL-RGHA architecture, showing the server-side RGPC and GHA modules.

Datasets

The framework was evaluated on five standard benchmarks across vision and speech tasks:

  • Vision: FMNIST, CIFAR-10, and CIFAR-100 using ResNet-18 and ResNet-32 backbones.
  • Speech: The Speech Commands dataset (keyword spotting) and the SNIPS dataset (intent classification).
  • Heterogeneity: Data was partitioned using a Dirichlet distribution ($Dir(\alpha)$) to simulate varying levels of statistical skew.

Key Insights and Ideas

  • Direction Over Magnitude: In non-IID settings, a gradient with a large magnitude often indicates local overfitting rather than a useful global update. The authors found that directional consistency is a much more reliable indicator of an update's utility.
  • Geometric Alignment vs. Naive Averaging: Traditional aggregation (like FedAvg) can lead to "gradient cancellation" where opposing updates zero each other out. By ensuring all gradients reside in the same half-space, GHA prevents destructive interference.
  • Server-Side Coordination is Key: The entire alignment process happens on the server. This preserves the core advantage of PSL—minimal communication overhead for the clients—while providing the stability typically only seen in synchronized federated methods.
Figure 6 validates that clients with higher consistency scores (left) indeed yield higher test accuracy (right), justifying the use of consistency for ranking.

Key Results

PSL-RGHA consistently outperformed existing baselines (standard PSL, SFL, and EPSL) across all tested scenarios:

  • Accuracy Boost: On non-IID CIFAR-10 with ResNet-18, PSL-RGHA achieved 82.68% accuracy, a massive improvement over standard PSL's 52.32%.
  • Robustness to Heterogeneity: While other methods saw sharp performance drops as data became more skewed, PSL-RGHA remained remarkably stable.
  • Scalability: The framework maintains high performance even as the number of participating clients increases (up to 40 in experiments), whereas naive aggregation methods suffer from the accumulation of gradient noise.
Figure 2 illustrates that as data heterogeneity increases (lower alpha), the gradient conflict ratio in standard PSL spikes, leading to a sharp drop in test accuracy.

Limitations and Future Work

While highly effective, the current PSL-RGHA framework focuses on synchronous training, where the server waits for all clients in a round. The authors suggest that extending this geometric alignment principle to asynchronous training scenarios would be a valuable next step. Additionally, they aim to explore the application of these techniques to the fine-tuning of large-scale foundation models, where resource constraints on edge devices are even more pronounced.

Summary updated

Abstract

The escalating complexity of deep neural networks hinders the deployment of federated learning (FL) on resource-constrained devices, especially for multimedia applications such as on-device image classification and speech recognition. Parallel split learning (PSL) has emerged as a promising solution by offloading a substantial portion of the computational workload to the server through layer-wise model partitioning, making it particularly suitable for resource-limited visual and speech intelligence scenarios. However, the absence of client-side aggregation renders PSL sus ceptible to severe training divergence and potential convergence failure under heterogeneous data distributions. To address this challenge, we propose PSL-RGHA, a gradient-aligned PSL framework that employs reference gradient half-space alignment mechanism to mitigate training divergence among client devices. PSL-RGHA comprises two novel components: reference gradient progressive construction (RGPC) and gradient half-space alignment (GHA). Specifically, RGPC evaluates the directional consistency of client gradients and progressively synthesizes a robust reference gradient via Weiszfeld updates, effectively approximating the global descent direction. Then, GHA constrains client gradients within the reference half-space defined by RGPC to mitigate gradient conflicts and rectify local updates. Extensive experiments demonstrate that our PSL-RGHA framework outperforms existing baselines in various settings.
BibTeX
@misc{li2026psl,
  author = {Han Li and Zheng Lin and Miao Yang and Zehang Lin and Jing Yang and Zihan Fang and Qiyang Zhang and Zhaolu Kang and Jianmin Li and Lin Li and Mingwei Hong and Shunzhi Zhu},
  title = {{PSL-RGHA: Parallel Split Learning with Reference Gradient Half-Space Alignment for Mitigating Gradient Divergence}},
  year = {2026},
  eprint = {2611.14312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14312},
}
Figure