In split learning, a phone or sensor runs the first few layers of a network and ships the intermediate activations to a server that runs the rest. Training this way is slow because of a "ping-pong" wait. On every step the device must sit idle until the server sends back the gradient for its activations. Recent systems remove the wait by giving each device a tiny local model that guesses that gradient, and they correct the guesser on a fixed schedule. The schedule is the weak point. Each device either talks to the server every $h$ steps or realigns at fixed intervals, no matter whether its current data is familiar or new. A static schedule wastes communication on easy batches and lets the guesser drift on hard ones.
We propose to make the guesser report how unsure it is and to contact the server only when that uncertainty is high. The technique is old: run a few dropout-perturbed copies of a model and read the spread of their outputs as a measure of ignorance. It is usually rejected as too expensive because it multiplies inference cost by the number of samples. That cost matters when the network is large. Here it is paid only on the small gradient-guessing head, so the price is a few extra passes through a model that is already a rounding error next to the device's own layers. In return, the device can skip most of the blocking round trips that dominate training time.
Core Claim
Sampling-based epistemic uncertainty is affordable when it is confined to the tiny auxiliary gradient estimator. Used as a per-batch synchronization gate, it should reach the accuracy of frequent synchronization at the communication cost of infrequent synchronization. It should also beat both static schedules and cheaper deterministic gates, specifically when a client's data shifts away from what the estimator has already been corrected on.
Background and Positioning
Three works define the latency problem and supply the machinery.
SplitFed runs clients in parallel and averages their device-side layers. This removes the need for clients to wait for one another, but each client still blocks on every step for the server's gradient.
CSE-FSL breaks the per-step wait by attaching a local loss head to each client and uploading only every $h$ steps. Its own numbers motivate this proposal. On CIFAR-10, $h=50$ cuts total traffic from about 172 GB to 2.69 GB, but accuracy drops from 77.96% at $h=1$ to about 73%.
FSL-SAGE goes further. Its tiny client models estimate the server's gradient with respect to the activations directly, and the server realigns them periodically. It reports 81% accuracy on CIFAR-10 with 2.2× less communication than CSE-FSL and 10× less than SplitFed. Its alignment interval is still fixed, and its estimator has no notion of its own confidence.
We reuse the FSL-SAGE estimator unchanged and replace only the decision of when to synchronize.
The uncertainty side supplies the tool and also constrains its use.
Dropout as a Bayesian Approximation supplies the estimator. Keeping dropout on at test time and taking the spread across passes approximates model uncertainty. On rotated MNIST, that spread rises even while softmax confidence stays high. We use it on a head of a few thousand parameters rather than a full network.
AdaNSP motivates the control idea. It uses dropout variance to decide when a parser should spend extra computation, and removing that mechanism costs about 2.8 to 2.9 points. We transfer the same "spend effort where uncertain" logic from computation inside one model to communication across a network link.
UPS offers further support. Filtering by uncertainty alone cut CIFAR-10 error from 16.50% to 9.46% compared with confidence-only selection, which shows that variance can catch errors that confidence misses.
Two papers constrain the design.
Selective Classification for Deep Neural Networks found that plain softmax confidence can rank errors as well as or better than MC-dropout on ImageNet. A deterministic gate is therefore a real competitor, and we treat it as one.
BESplit is the closest bridge between the two fields. It computes single-pass evidential uncertainty on uploaded activations at the server, using it to weight client aggregation and correct non-IID bias, with under 0.2% runtime overhead. It does not use uncertainty to decide when a client may train without the server, and it never touches the gradient estimator. Our contribution is orthogonal and could be stacked on it.
Research Program
Each client holds its device-side layers and an FSL-SAGE-style auxiliary head $f_\phi$ with dropout. The head maps a batch of activations $a$ to an estimated activation gradient. At each step the client draws $N$ dropout masks, obtaining estimates $g_1,\dots,g_N$ with mean $\bar g$. It then computes a relative dispersion
$$u(a) = \frac{1}{N}\sum_{i=1}^{N} \frac{\lVert g_i - \bar g\rVert^2}{\lVert \bar g\rVert^2}.$$
If $u(a) \le \tau_t$, the client backpropagates $\bar g$ locally and moves on. If $u(a) > \tau_t$, it uploads $a$ and the labels, waits for the true gradient, uses that gradient for the step, and adds the pair to its alignment buffer for $f_\phi$.
The threshold $\tau_t$ is a running quantile of recent $u$ values, chosen so that the long-run synchronization rate equals a target $\rho$. This makes every comparison exact at matched communication: $\rho = 1/h$ spends the same bytes as CSE-FSL with step count $h$.
We will also test a non-blocking variant. There, the client proceeds with $\bar g$ even on triggered batches and uses the returned true gradient only to realign $f_\phi$. This trades some gradient accuracy for zero waiting.
The analysis has three parts.
First, we measure whether $u$ actually predicts the true gradient error $\lVert \bar g - g^\star\rVert / \lVert g^\star \rVert$. We will collect the true gradient $g^\star$ on a held-out fraction of steps purely for evaluation.
Second, we test whether gated synchronization, at a fixed $\rho$, lowers the cumulative error of the gradients actually applied, compared with periodic synchronization. If it does, the FSL-SAGE $O(1/\sqrt{T})$ argument should carry over with a smaller bias term. We will attempt that bound but treat it as secondary.
Third, we measure server load. Gating thins the stream of activations the server must process, which bears on the centralized queue that single-server designs face when many clients train at once.
Baselines
The naive reference point is vanilla SplitFed with blocking synchronization on every step, following SplitFed. It sets the accuracy ceiling and the communication and latency floor. We run it with the same backbone, cut layer, client count, and data partition as every other method.
The first static competitor is CSE-FSL at $h \in \{1, 10, 50\}$. Its reported CIFAR-10 points are 77.96% at $h=1$ and about 73% at $h=50$ with 2.69 GB of traffic. Beating it means exceeding its accuracy at $\rho = 1/h$ with equal bytes.
The strongest existing method is FSL-SAGE, run with its periodic alignment and its Lazy variant. Its reported result is 81% CIFAR-10 accuracy at 2.2× less communication than CSE-FSL. We will reproduce it in our own harness rather than compare against the paper's number. Its alignment interval gets the same tuning budget as our quantile target, with a grid of equal size. The gated method beats it if, at every matched byte budget on the accuracy–communication curve, it reaches higher final accuracy or reaches FSL-SAGE's final accuracy in fewer round trips. The gap must be larger than seed-to-seed spread across five seeds, and we expect it to grow under non-IID partitions.
Three ablations isolate the key ingredient. Random gating at the same $\rho$ tests whether the choice of which batches to synchronize matters at all. Deterministic gating uses a single-pass signal: either the auxiliary head's predicted gradient norm or the entropy of the client's local loss head, as in CSE-FSL. A learned error head is a second small network trained on alignment pairs to regress the gradient error directly, at $O(1)$ cost. These ablations share the head architecture, $\rho$, and tuning budget. They separate "sampling-based epistemic uncertainty" from "any confidence signal."
Why not just…?
Why not just keep a static $h$ and tune it? CSE-FSL's own curve shows the cost: about five accuracy points between $h=1$ and $h=50$ on CIFAR-10. A fixed schedule cannot spend its budget on the batches where the guesser is wrong. The quantile-matched comparison settles this directly. If gated and periodic synchronization tie at equal $\rho$ on non-IID splits, the idea fails.
Why not just use the head's own confidence, a single deterministic pass? This is a real threat. Selective Classification shows softmax confidence often ranks errors well. A deterministic signal does reflect familiar difficulty. It is less likely to reflect novelty, which is the failure the MC-dropout paper documents on rotated digits. The early correlation study will tell us which signal predicts true gradient error, broken down by in-distribution batches versus batches from a client whose class mix the head has not been aligned on. If the deterministic gate matches ours on both, we will report it as the method. The finding becomes "confidence-gated synchronization," and the sampling component is dropped.
Why not just train a small network to predict the guesser's error? This is the strongest alternative, because it is cheap and directly targeted. Its weakness is structural. It learns only from alignment pairs already collected, so on genuinely new data it can be confidently wrong in the same way the gradient guesser is. We will include it as a co-equal gate and predict that it matches ours under IID data but falls behind after a mid-training shift in a client's data. If it does not fall behind, it wins, and we will say so.
Why not just use evidential uncertainty as BESplit does? BESplit applies evidential uncertainty to class predictions at the server, not to a regression over gradients on the client. Evidential formulations for vector-valued gradient targets would need new regularizers with the tuning sensitivity the file flags. We will treat an evidential gradient head as an optional fourth gate rather than a default.
Proposed Experiments
Image classification. Main runs use CIFAR-10 and CIFAR-100 with ResNet-18 split after the early blocks. We use 10 and 100 clients, with IID partitions and Dirichlet non-IID partitions. For context, Mix2SFL reports about 67% on CIFAR-10 at 100 clients, which shows how hard that regime is.
Natural client heterogeneity. F-EMNIST tests heterogeneity that arises naturally from different writers. HAM10000 tests high-stakes medical non-IID data, where SplitFed variants reach 76 to 79% accuracy.
Transformers. The E2E task with GPT2-medium and LoRA, as in FSL-SAGE, tests whether the gate carries over to Transformers. We found no reported scalar result on Lacuna for E2E, so we will compare only within our own runs.
Distribution shift. One deliberate shift experiment swaps half of each client's classes midway through training. This is where we predict the gap between sampling-based and deterministic gates will appear.
Metrics. We report final accuracy versus total bytes transferred, and the number of blocking round trips. We report wall-clock time-to-accuracy under simulated uplink latencies of 10, 50, and 200 ms. We report server examples processed per round, and the added client floating-point operations from the $N$ dropout passes, with $N \in \{4, 8, 16\}$.
Decisive checks. The first check comes before any training comparison. On frozen FSL-SAGE checkpoints, the rank correlation between $u$ and the true gradient error must clearly exceed that of the deterministic signals, especially on shifted batches. Otherwise the gate has nothing to work with.
The second check is the matched-$\rho$ accuracy comparison against FSL-SAGE and random gating on non-IID CIFAR-10. The third is that the added sampling cost on the client stays under a few percent of its per-step compute. If this cost does not stay small, the premise that stochastic overhead is cheap when confined to the head is false.
References
- SplitFed: When Federated Learning Meets Split Learning
- Communication and Storage Efficient Federated Split Learning
- FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- AdaNSP: Uncertainty-driven Adaptive Decoding in Neural Semantic Parsing
- In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
- Selective Classification for Deep Neural Networks
- BESplit: Bias-Compensated Split Federated Learning with Evidential Aggregation
- Mix2SFL: Two-Way Mixup for Scalable, Accurate, and Communication-Efficient Split Federated Learning