Split Deep Kernel Learning Through a Factorized Cut Layer

Many useful regressors need honest error bars as well as good predictions. A Gaussian process (GP) is a Bayesian model that predicts a mean and a variance for every input, and it provides both. Some of these regressors must also be trained across devices that cannot share their raw data and cannot run a full network. Split learning handles the compute and privacy side. It cuts a network in two: the device runs the first layers, sends the intermediate activations to a server, and the server runs the rest. Gaussian processes handle the uncertainty side, but only once they are made cheap. The standard way to do that is random Fourier features, which replace the kernel with a finite, explicit feature vector so that the GP becomes Bayesian linear regression on $m$ features. We propose putting these pieces together so that each one removes a limitation of another. The device computes the thin half $V$ of a low-rank factorized cut layer $W \approx UV$ and transmits only an $r$-dimensional code. The server applies $U$, the remaining layers, and a random-feature GP head. The whole system is trained end to end on the GP's marginal likelihood, which is the probability the model assigns to the observed training targets.

The key observation is structural. The marginal likelihood does not split into a sum of per-sample terms. It depends on global summary statistics: the $m \times m$ feature scatter matrix and the feature–target products. Federated GP methods therefore cannot optimize it through the feature network. Each client sees only its own data, which is why Federated Bayesian Neural Regression (FedBNR) trains its feature network on an ordinary loss first and fits the GP afterwards with the network frozen. In split learning the server already sees every client's activations. It can maintain the global statistics itself and send exact-in-expectation marginal-likelihood gradients back across the cut. The claim is that split topologies make end-to-end deep kernel learning feasible under data locality. The width $r$ of the factorized cut is then a single dial that trades upstream bandwidth against predictive calibration, and we will measure that trade-off directly.

Core Claim

The setup is as follows. The server-side network maps the received code to a representation $h$. Random Fourier features turn $h$ into $\phi(x) \in \mathbb{R}^m$, and the GP is Bayesian linear regression $y = w^\top \phi(x) + \epsilon$ with prior $w \sim \mathcal{N}(0, s^2 I)$ and noise variance $\sigma^2$. Write $S = \sum_i \phi_i \phi_i^\top$ and $b = \sum_i \phi_i y_i$, where both sums run over all clients' data. Write $A = S + (\sigma^2/s^2) I$. The log marginal likelihood is then

$$\log p(y) = -\frac{1}{2\sigma^2}\left(\sum_i y_i^2 - b^\top A^{-1} b\right) - \frac{1}{2}\log|A| + \frac{m}{2}\log\frac{\sigma^2}{s^2} - \frac{n}{2}\log(2\pi\sigma^2),$$

where $n$ is the total sample count. Once $(S, b)$ is fixed, the gradient with respect to any single $\phi_i$ depends only on $\phi_i$, $y_i$, $A^{-1}$ and $A^{-1}b$. The server can therefore compute per-sample gradients using running global statistics and backpropagate them through $U$ to each client's $V$ and early layers. We claim three things. First, this end-to-end training gives better-calibrated predictions than the two-phase FedBNR recipe on non-i.i.d. client splits. Second, a factorized cut trained this way reaches a given negative log-predictive density at a much smaller transmitted width $r$ than an unfactorized cut compressed by quantization. Third, the marginal-likelihood objective reduces or removes the need for the full-rank warm-up that factorized training normally requires. That warm-up is infeasible on memory-limited clients.

Relation to Prior Work

Four papers each cover part of this design, and none combines the parts. SplitFed supplies the topology. A main server runs the upper layers for many clients in parallel while a second server averages client weights. It produces deterministic outputs and transmits uncompressed activations. We reuse its topology and replace both the output head and the cut. Deep Kernel Learning motivates the objective. It shows that backpropagating the GP marginal likelihood into the network matters. However, it is centralized, and its grid-based interpolation limits the final feature dimension to about five. Random features remove that limit, and the split setting removes centralization. How Good are Low-Rank Approximations in Gaussian Process Regression? constrains the design. It proves that truncated Mercer expansions beat random features at equal rank, especially for rough Matérn kernels. We take random features anyway because Mercer eigenfunctions require the input distribution, which is fragmented across clients. We treat that paper's centralized Mercer GP result as an upper reference, not a competitor we expect to beat. FedBNR is the closest work. It already uses random features and scatter-matrix aggregation to get a global primal GP. However, its clients run the full dense extractor and it trains in two disjoint phases. We move most of the extractor to the server, factorize the interface, and train jointly.

On the factorization side, Pufferfish factorizes layers before data-parallel training to cut gradient traffic. It finds that it needs a full-rank warm-up and must keep boundary layers full-rank. Reduced order modelling of latent features in LLMs reaches the opposite conclusion for later layers, finding them the most compressible. A split cut is simultaneously the client's last layer and the server's first. Whether it behaves as a fragile boundary or a compressible latent is exactly what our rank sweep will reveal. These two papers supply the method, SVD-based factor initialization, and the open tension.

Research Program

We will build the system on the SplitFed topology with MLP extractors for tabular regression. The client holds the input layers and $V \in \mathbb{R}^{r \times d}$. The server holds $U$, the deeper layers, and the random-feature head with learnable lengthscales. The main server keeps exponentially averaged estimates of $S$ and $b$, updated from each client batch's features. It also recomputes them exactly with a full forward pass every $T$ steps to bound staleness. Per-sample gradients use the current $A^{-1}$, obtained by a Cholesky factorization of an $m \times m$ matrix. For $m$ in the low thousands this is cheap on the server, and the cost does not depend on $n$. After training, the posterior mean and variance are closed form from the final exact statistics.

The analysis has three parts. First, we will bound the gradient bias caused by stale statistics as a function of $T$ and the learning rate, and check the bound empirically against exact full-batch gradients on small data. Second, we will trace NLPD and RMSE as the cut rank $r$ shrinks, to see whether calibration fails before accuracy does. Third, we will compare three initializations of the factorized cut: from scratch, a full-rank warm-up with SVD initialization in the Pufferfish style, and a warm-up run only on the server using a small public proxy set. Since labels and all heavy computation sit on the server in our topology, we will also run a feature-inversion attack on the transmitted code at each $r$. This tests whether narrower cuts leak less, a side effect we suspect but do not claim.

Baselines

The naive reference point is deterministic SplitFed with an unfactorized cut and a mean-squared-error head. Its prediction intervals come from a single homoscedastic noise estimate. This shows how much the GP head adds at all. It is run with the same extractor, the same client partitions and the same number of optimizer steps.

The strongest existing method on this question is FedBNR's two-phase recipe, transplanted into the same split architecture so that only the training procedure differs. Phase one trains the split network with mean-squared error. Phase two freezes it and fits the random-feature posterior in closed form from aggregated scatter matrices. This is the strongest competitor because it is the only published method producing a global primal GP under data locality. It also isolates our key ingredient, end-to-end marginal-likelihood training. Both variants get the same random-feature count $m$, cut rank $r$, and hyperparameter search budget (the same number of trials over learning rate and lengthscale initialization). FedBNR reports that it beats FedAvg, FedProx and pFedGP in RMSE, with markedly lower expected calibration error on Skillcraft, SML, Parkinsons, Bike and CCPP. However, there is no reported numerical result for FedBNR on these datasets on Lacuna, so we will reproduce it ourselves. Beating it means lower test NLPD and lower expected calibration error, averaged over at least five non-i.i.d. partition seeds, with RMSE no worse.

The second strong competitor is a sparse variational inducing-point GP head on the server, trained end to end on the same split extractor. This is essentially Deep Kernel Learning with inducing points instead of grids. It is a fair and serious alternative because in split learning the server already sees features. The privacy objection from the file, that cross-client distances leak, does not apply here. It is run with a number of inducing points chosen so that the server memory and FLOPs per step equal those of our head. Beating it means comparable or better NLPD at equal server compute.

As an upper reference we report exact centralized GPs where $n$ allows. On Sarcos we compare against the centralized Mercer GP of rank 300, which How Good are Low-Rank Approximations in Gaussian Process Regression? reports at an NLPD of −0.45. We do not expect to beat it. We report how close a data-local split system comes.

Two further ablations isolate the factorized cut. The first keeps the cut unfactorized but quantizes activations to match the byte budget of the rank-$r$ code. The second places the factorization in a middle server layer instead of at the cut, holding total parameters fixed. The factorized cut wins if it dominates quantization on the NLPD-versus-bytes curve.

Why not just…?

Why not just use an inducing-point GP head on the server? This is a real threat. We cannot dismiss it on privacy grounds, because the server sees features either way. Random features have concrete advantages here: exact closed-form posteriors from $m \times m$ statistics that can be updated incrementally as clients join, and no inducing locations to optimize. Approximation-Aware Bayesian Optimization also reports that finite inducing budgets oversmooth sharp peaks. Against this, the Mercer-versus-random-features result suggests that random features may need a larger $m$. The equal-compute comparison above settles it. If inducing points win, the paper's contribution narrows to the claim that server-side global aggregation enables end-to-end GP training through a cut. That claim does not depend on the head.

Why not just train two-phase, as FedBNR does, and skip end-to-end marginal likelihood? If the frozen features from a mean-squared-error network already give well-calibrated posteriors, our main mechanism is unnecessary. Deep Kernel Learning and the Deep Fourier GP results in the low-rank-approximation paper both argue that joint training matters, but only in centralized settings. The FedBNR-in-split ablation tests this directly. If the gap is within seed noise on the non-i.i.d. UCI partitions, we will report that negative result and reframe the work around the rank-bandwidth analysis alone.

Why not just compress the activations with quantization or an autoencoder instead of factorizing the cut? Quantization adds no parameters and needs no redesign. A factorized cut, however, also shrinks the client's compute and memory, since the client only stores $V$. It also shapes the transmitted code during training rather than distorting a code afterwards. The byte-matched quantization ablation shows which effect dominates. An autoencoder at the cut would add client-side layers, which undercuts the resource constraint that motivates splitting.

Why not just add conformal prediction to a deterministic split network? Conformal methods such as those in Approximating Full Conformal Prediction with Gauss-Newton Influence guarantee marginal coverage but give no pointwise posterior variance. Acquisition functions in Bayesian optimization, as in BOHB, need that variance. We include split conformal on the naive baseline and report interval coverage and width next to our GP intervals, so readers can judge the cost of Bayesian machinery when coverage is the only goal.

Proposed Experiments

The main testbed uses the FedBNR datasets (Skillcraft, SML, Parkinsons, Bike, CCPP) under that paper's heterogeneous partition protocol, plus Elevators and 3DRoad. For scale we use Electric (about 2 million points) and Airline (about 5.9 million). Large-Scale Learning with Fourier Features and Tensor Decompositions reports lower MSE than two variational GP baselines on Airline and about 26 minutes of training on a laptop at rank 5. The file gives no exact MSE for that paper, so we will rerun it as a centralized accuracy reference. We use Sarcos for the centralized NLPD reference described above. Metrics are test NLPD, RMSE, expected calibration error and 95% interval coverage, together with upstream and downstream bytes per sample, client peak memory, and wall-clock time per epoch.

Three checks are decisive. The first is the end-to-end versus two-phase comparison at matched $r$ and $m$. The paper's central claim stands or falls on a consistent NLPD and calibration gain there. The second is the rank sweep over $r$ from 2 up to the full width. It tests whether the factorized cut dominates byte-matched quantization, and whether calibration degrades at a larger $r$ than accuracy does. Such a result would tell practitioners that bandwidth budgets should be set by uncertainty quality, not by RMSE. The third is the initialization study. If from-scratch factorized training under the marginal likelihood matches the Pufferfish-style warm-up within noise, constrained clients never need to hold a full-rank layer, which resolves the warm-up tension directly. As a secondary check, we measure the bias from stale statistics against exact gradients as the refresh interval $T$ varies, to confirm that the running-estimate shortcut does not account for any observed gains or losses.