Host-Memory Preconditioners for Exact Gaussian Process Training

Exact Gaussian processes are flexible models that give calibrated uncertainty. At a million training points, the best current way to train them is to solve linear systems with conjugate gradients, an iterative method that needs only products of the kernel matrix with vectors. Those products are recomputed on the fly on GPUs so the $n \times n$ matrix is never stored. The remaining cost is the number of iterations. Every iteration is a full $\mathcal{O}(n^2)$ pass, and the number of iterations depends on the quality of the preconditioner, a cheap approximate inverse that makes the system easier to solve. On GPUs the preconditioner's size is capped by device memory, which the kernel products already crowd. Meanwhile, the host machine's dozens of CPU cores and hundreds of gigabytes of RAM sit idle for the whole run.

We propose moving the preconditioner to the host. The CPU builds, stores and applies a low-rank preconditioner roughly ten times larger than fits on the GPU. The GPU does only what it is best at, the matrix-free kernel products. The two devices work at the same time on independent halves of the batched right-hand sides. The bet is that the CPU is most useful when it is given a job that cuts the number of expensive GPU passes, not when it contributes its few spare FLOPs to those passes. This split also has an unusual numerical property. A preconditioner computed from slightly out-of-date hyperparameters does not change the answer, only the speed. The CPU can therefore rebuild it asynchronously without the consistency problems that pipelined co-processing usually creates.

Core Claim

Training an exact Gaussian process means repeatedly solving $(\hat K)^{-1}\mathbf{y}$ and estimating $\log|\hat K|$ and its gradients, where $\hat K = K + \sigma^2 I$ is the $n \times n$ kernel matrix plus noise. In the approach of GPyTorch and Exact Gaussian Processes on a Million Data Points, each solve is preconditioned by a rank-$k$ pivoted Cholesky factor $L \in \mathbb{R}^{n\times k}$, with $P = LL^\top + \sigma^2 I$. Applying $P^{-1}$ via the Woodbury identity costs $\mathcal{O}(nk)$ per vector, which is cheap relative to the $\mathcal{O}(n^2 d)$ kernel product. What limits it is memory. At $n = 1.3\times 10^6$ and $k = 5000$, $L$ occupies about 26 GB in single precision, which is incompatible with a 16–32 GB GPU that also holds kernel tiles.

Preconditioning for Scalable Gaussian Process Hyperparameter Optimization showed that the preconditioner should also be used in the log-determinant:

$$\log|\hat K| = \log|P| + \log|P^{-1}\hat K|,$$

where the first term is exact and cheap and only the second is estimated by stochastic trace estimation, which averages random probe vectors. A better $P$ therefore cuts two costs: the number of conjugate-gradient iterations, and the number of probe vectors needed for a given gradient noise level. Both costs are paid in full-kernel passes.

Our claim is that on a single node with a large host, raising $k$ by an order of magnitude on the CPU reduces total full-kernel passes enough to beat pure-GPU exact training in wall-clock time at equal accuracy, and to do so by more than naively splitting kernel tiles between CPU and GPU. The heuristic reasoning behind the claim is that the preconditioned condition number is governed by the trailing kernel eigenvalues $\lambda_{k+1}, \lambda_{k+2}, \dots$ relative to $\sigma^2$, and CG iterations scale with its square root. The benefit therefore depends on how fast the spectrum decays past the GPU-feasible rank. That is an empirical question per dataset, and it is the decisive one.

Research Program

We would build three pieces, each tested in isolation.

The first is a host-resident pivoted Cholesky that runs in double precision on CPU cores. Pivoted Cholesky selects its next pivot greedily and one at a time, with scalar square roots and divisions. Exploiting the capabilities of modern GPUs for dense matrix computations found exactly these operations underuse GPUs, and it got its hybrid Cholesky speedups by routing them to the host while the GPU did bulk products. That paper supplies the design principle we reuse, but it factored explicitly stored dense matrices. Our factor is instead built column by column from coordinates, with kernel columns generated on demand.

The second is a split batched solver. The modified batched conjugate gradient of GPyTorch solves many right-hand sides at once: $\mathbf{y}$ plus the probe vectors. These systems are mathematically independent, so we split them into two groups and run them out of phase. While the GPU computes the kernel product for group A, the CPU applies $P^{-1}$ to group B, and then they swap. The only data crossing the PCIe bus are the vectors themselves: at $n = 1.3$M with 32 probes, about 170 MB in single precision per iteration. Moving that takes tens of milliseconds, against multi-second kernel passes. No kernel matrix is ever transferred. Unlike the bounded-staleness pipelining of NeutronOrch, this overlap is exact, because no computation reads stale data.

The third is an asynchronous preconditioner refresh across hyperparameter steps. When the optimizer moves from hyperparameters $\theta_{t-1}$ to $\theta_t$, the GPU starts solving at $\theta_t$ immediately using $P$ built at $\theta_{t-1}$, while the CPU rebuilds $P$ at $\theta_t$ for the next step. This is where our design departs from the heterogeneous literature's usual concern. The research file flags staleness as hard to reconcile with numerical optimization. Here, any fixed symmetric positive-definite $P$ leaves the conjugate-gradient solution exact, and the log-determinant decomposition above stays an identity: $\log|P|$ is simply constant in $\theta_t$, so its gradient contribution vanishes and the trace term remains an unbiased target. Staleness only costs iterations. We will measure that cost directly as a function of the hyperparameter step size, and fall back to synchronous rebuilds when the step is large, for example early in optimization.

We would also try, as a secondary piece, mixed precision in the style of Barrachina et al.: single-precision kernel products on the GPU, with the preconditioner and the running solution accumulated in double precision on the host. We treat this as optional because GPyTorch already reports that CG is robust in float32.

Relation to Prior Work

The direct motivation comes from two exact-GP systems. Exact Gaussian Processes on a Million Data Points partitions kernel products across eight GPUs and keeps the host as a coordinator. GPyTorch keeps both products and preconditioner on the device. We keep their solver and change where the preconditioner lives and how large it can be.

The GP-specific heterogeneous work is related in hardware but different in what it runs. Scaled Block Vecchia uses CPUs for neighbor search and GPUs for batched block Cholesky, sequentially, and its answer is approximate by construction. Dai et al. distribute sparse variational statistics over CPU nodes and GPUs, also sequentially. Neither co-processes a Krylov solve for the exact model.

The general co-processing results constrain what we should expect from the obvious alternative. The roughly 16% gain from giving the CPU a share of the same kernel in RNA Folding Algorithms shows that sharing homogeneous work is worthwhile but modest. The layer-wise asymmetric split in NeutronOrch and the host-side coarse-grid handling in hybrid topology optimization show the larger wins come from giving the CPU a structurally different, memory-hungry task. That is the analogue we pursue.

Baselines

The naive reference point is GPyTorch's exact GP as used by Wang et al.: matrix-free partitioned kernel products and a GPU-resident pivoted Cholesky preconditioner at the default rank. We run it on the same node, one or two GPUs plus a many-core host, with the host idle. Beating it means lower wall-clock time to a fixed marginal-likelihood tolerance, with test RMSE and negative log-likelihood unchanged within seed variance.

The strongest existing method is the same pipeline plus the preconditioned log-determinant estimator and L-BFGS of Preconditioning for Scalable Gaussian Process Hyperparameter Optimization, which reports up to a 12x wall-clock speedup. We configure it with the largest preconditioner rank that fits in GPU memory beside the kernel tiles, and with the same probe count, tolerance, optimizer and iteration budget as our method. Our method must beat this configuration, not a weakened one.

For accuracy reference, the file's only quantitative figures for the strongest exact system are Kin40K test RMSE of 0.099 (against SGPR's 0.273) and HouseElectric training in under two hours on eight V100s, both from Wang et al.. No reported result was found on Lacuna for per-solve CG iteration counts as a function of preconditioner rank on any of these datasets. That curve is something we must measure ourselves.

A third baseline is naive co-processing: the kernel product tiles are split between CPU and GPU in proportion to measured throughput, with no preconditioner change. It tests whether the CPU's FLOPs are better spent on the product or on the preconditioner.

Two ablations isolate the key ingredient. The first keeps rank fixed at the GPU-feasible value but places it on the host, which separates the effect of offloading from the effect of larger rank. The second uses large rank with synchronous rather than asynchronous refresh, which isolates the cost and benefit of staleness. All methods get the same hyperparameter initialization, including Wang et al.'s subset pretraining when it is used, the same random seeds and the same stopping rule.

Why not just…?

Why not just split the kernel products between CPU and GPU? It is the obvious co-processing move, and it is guaranteed to help a little. Its ceiling, though, is the ratio of host to device throughput on a compute-bound dense kernel, which puts it in the regime of the 16% gain reported in RNA Folding Algorithms. It also does nothing about the number of passes. We include it as a baseline. If it matches our method, the preconditioner story is wrong.

Why not just add GPUs and shard the large preconditioner across them? This is the real threat. With eight GPUs, a 26 GB factor fits. In that setting our method's advantage reduces to cost and energy, which Scaled Block Vecchia emphasizes by reporting its consumption as 12–40% of exact solvers. It would no longer be a speed advantage. We will therefore report two comparisons. One is time-to-accuracy on a fixed node. The other is time and energy per GPU-hour against a multi-GPU sharded preconditioner, so the reader can see where host memory stops being the better place for it.

Why not just use a sparse variational GP? It fits on one GPU trivially. However, Wang et al. report a Kin40K test RMSE of 0.099 for the exact GP versus 0.273 for SGPR, and they report that sparse methods saturate on Song as $n$ grows. The problem we target only exists for users who need the exact model, and we will report the SVGP gap on our datasets to show it persists.

Why not just use a fast multipole product? Giga-scale Kernel Matrix-Vector Multiplication on GPU gives linear-time products, but only for dimension up to about seven. On 3DRoad ($d=3$) it is the right tool, and we concede that regime. Our method is aimed at Song ($d=90$) and similar moderate- and high-dimensional data.

Proposed Experiments

Datasets are the UCI suite used by the exact-GP papers: Kin40K, Elevators, Bike, Protein, KEGGdir, 3DRoad, Song and HouseElectric. They span $n$ from about $10^4$ to $1.3\times 10^6$ and dimension from 3 to 90.

The first and decisive experiment runs before any systems work. For each dataset, at hyperparameters fitted on a subset, we measure CG iterations to tolerance and trace-estimator variance as preconditioner rank grows from about 100 to about 10,000. If the curves flatten before the GPU-feasible rank, the project stops. This is plausible on high-dimensional data with slow eigenvalue decay, and the preconditioning paper's convergence guarantees are limited to univariate RBF kernels, so we cannot assume the curves will keep dropping.

If the curves keep dropping, the main experiment reports end-to-end wall-clock time to a fixed marginal-likelihood tolerance, total kernel-product passes, test RMSE and negative log-likelihood, CPU and GPU utilization, and node energy. These are measured for the baselines and ablations above on a node with one or two GPUs and a large-memory host. We then run HouseElectric on eight GPUs to compare against the published under-two-hour figure. A final stress test measures how many extra iterations a stale preconditioner costs as the hyperparameter step size grows. It checks that asynchronous refresh never changes the final solution beyond solver tolerance, confirming that staleness affects only speed.