Sparse Gaussian processes (GPs) are probabilistic regression models that summarize a large dataset with a few hundred "inducing points": learned pseudo-inputs that carry the model's knowledge. Training one means optimizing three different kinds of parameters at once. The first is a Gaussian distribution over function values at the inducing points. The second is the locations of those points. The third is the kernel settings, such as lengthscales and noise. Practitioners currently hand-tune a separate learning rate for each group. This is fragile, because the groups differ widely in gradient scale and coupling. It is also expensive when the GP must be refit thousands of times, for example inside Bayesian optimization.
A separate line of optimization work removes learning rates by estimating them from the trajectory. The rule is roughly "distance traveled from the start, divided by the accumulated size of the gradients." That work measures distance in plain Euclidean terms. We propose to measure, for each GP parameter group, how far it has moved in the geometry where that movement actually matters, and to set that group's step size from this distance. For the variational distribution the measure is the KL divergence from its starting point. For inducing points it is displacement counted in lengthscales. For kernel settings it is displacement in log-space. The result would be a sparse GP trainer with no learning rates, including for natural-gradient updates on non-Gaussian likelihoods. For those updates the step schedule is currently chosen by hand.
Core Claim
Distance-over-gradient rules are provably tuning-free when "distance" and "gradient size" are measured in the same norm. In sparse variational GPs the Euclidean norm on raw parameters is the wrong one. Euclidean steps on the variational distribution are badly conditioned: Natural Gradients in Practice reports Adam failing to reach the optimum on Naval after thousands of iterations while natural gradient descent gets there almost immediately. Moving an inducing point by 0.1 means nothing until you know the lengthscale.
Our claim is that a per-group rule of the form
$$\eta_t^{(g)} = \frac{\bar r_t^{(g)}}{\sqrt{\sum_{i \le t} \|h_i^{(g)}\|_{(g)}^2}}, \qquad \bar r_t^{(g)} = \max\Big(r_\epsilon,\ \max_{i\le t} d_g(\theta_i^{(g)}, \theta_0^{(g)})\Big)$$
will train sparse GPs to the quality of a tuned hybrid optimizer with no per-dataset tuning. Here $g$ indexes the parameter group and $d_g$ is that group's distance. The vector $h_i^{(g)}$ is the update direction at step $i$, measured in the norm dual to $d_g$, and $r_\epsilon$ is a tiny initial distance guess. A Euclidean version of the same rule should not reach that quality.
For the variational distribution $q(u)$ we set $d = \sqrt{2\,\mathrm{KL}(q_i \| q_0)}$. Locally this equals the Fisher-metric length of the displacement. The update is the natural gradient $\tilde g = F^{-1} g$, where $F$ is the Fisher information of $q$ and $g$ the ordinary gradient. Its size is measured as $\|\tilde g\|_F^2 = g^\top F^{-1} g$, so numerator and denominator live in the same geometry.
The new element is a step-size rule tied to information geometry. This is what Convergence of projected stochastic natural gradient variational inference leaves open: it treats natural-gradient variational inference as mirror descent under Bregman divergences, but only with pre-set step and batch schedules. The second new element is carrying the same rule over to inducing points in lengthscale units.
Research Program
The method reuses three existing pieces. From DoWG Unleashed and D-Adaptation we take the distance-over-gradient step rule and its weighted-averaging variants. From Natural Gradients in Practice we take the hybrid structure: natural gradients for $q(u)$, first-order updates for inducing locations $Z$ and kernel settings. From the projected natural-gradient paper we take a projection step that keeps the covariance of $q(u)$ positive definite. That projection matters here. Distance-scaled rules tend to inflate the step until the iterates start to destabilize. For $q(u)$, that behavior would otherwise produce indefinite covariances rather than merely noisy steps.
The first work package is the variational block. A Gaussian $q(u)$ has a closed-form KL divergence at $O(m^3)$ cost, the same cost as the rest of the $O(Bm^2 + m^3)$ iteration, where $m$ is the number of inducing points and $B$ the minibatch size. With a Gaussian likelihood and full batch, a natural-gradient step of size 1 lands exactly on the optimum. This gives a sharp sanity check: the rule should grow from $r_\epsilon$ toward 1 within a few iterations. That growth reproduces the rising schedule Salimbeni et al. recommend, but driven by data rather than chosen by hand. We will test whether the rule needs a cap at 1 or whether gradient accumulation alone keeps it stable.
The second work package covers inducing locations and kernel settings. For $Z$ the distance is $\|(Z_i - Z_0)\Lambda^{-1}\|_F$, where $\Lambda$ holds the current lengthscales, and the gradient is rescaled by $\Lambda$ to match. Kernel settings use log-space displacement.
The central risk is the one flagged in the research file: an aggressive step can drive inducing points together until the inducing-point covariance matrix loses rank and its Cholesky factorization fails. We will log conditioning and failure counts throughout. The third work package is analysis. For the variational block with the other groups fixed, we will extend the DoG-style argument to mirror descent with a KL-derived distance. The target is a bound matching the tuned-schedule rates of the projected natural-gradient paper up to logarithmic factors.
Baselines
The naive reference is ordinary Adam at its common default rate of 0.01 on all parameters, with no grid search. This is what many practitioners actually run, and it shows how much tuning buys.
The strongest existing method is the hybrid of Natural Gradients in Practice: natural gradients for $q(u)$ with the authors' increasing step schedule, and Adam for $Z$ and kernel settings. We will give it a per-dataset grid over the Adam rate and the natural-gradient schedule's endpoints. Its cost is charged as the total compute of the grid, and it is also reported at its best single configuration. We found no reported numerical ELBO, NLPD or RMSE for this hybrid on Lacuna. The paper reports qualitatively that on Naval natural gradient descent converges "almost immediately" where Adam fails after thousands of iterations, at about 1.5x the cost per step. On the regression sets, CGLB reports outperforming SGPR on RMSE and NLPD across bike, elevators, poletele, kin40k and protein, but no per-dataset numbers were found on Lacuna. We will therefore compute all reference values ourselves under a shared protocol. To beat the hybrid, the method must match its best-configuration test NLPD within run-to-run noise on at least four of the six regression sets while using one run instead of a grid. It must also reach within a fixed ELBO tolerance in comparable or fewer iterations.
The off-the-shelf parameter-free competitor is Prodigy or D-Adapted Adam applied to all parameters with one global Euclidean distance. The key ablation replaces the KL and lengthscale distances in our rule with Euclidean distances on the same coordinates, keeping per-group accumulation. A second ablation uses one global distance with our geometries. All methods share the same data splits, inducing-point initialization, minibatch size, iteration budget and hardware.
Why not just…?
Why not just run Prodigy or D-Adapted Adam on everything? This is the real threat. If it matches our method, the contribution collapses into "existing optimizers already work for GPs," which would itself be worth reporting. The reason to expect a gap is that these methods rest on Adam's diagonal Euclidean preconditioning, which the Naval result shows is the failure mode on ill-conditioned variational problems. The research file also notes that D-Adaptation's distance estimates grow slowly or erratically in sensitive early phases. The Naval experiment and the Euclidean ablation settle this directly.
Why not keep the hand-designed natural-gradient schedule and just grid-search Adam? On a single dataset, this will probably do about as well as our method. The case against it is cost and brittleness under repeated refits. In Approximation-Aware Bayesian Optimization the surrogate is retrained across up to 80,000 evaluations and cannot be tuned per refit. Our Bayesian-optimization experiment measures exactly this.
Why not use a line search? Minibatch ELBO estimates are noisy, so sufficient-decrease tests compare inconsistent function values. DoWG Unleashed notes line searches also cost extra function evaluations per step. We include a full-batch backtracking variant only on the smallest dataset, as a reference for what a line search achieves without noise.
Proposed Experiments
For regression with minibatch sparse variational GPs, we use bike, elevators, poletele, kin40k and protein from the CGLB suite, plus Naval as the ill-conditioned case. Each is run with a Gaussian likelihood and with a Student-t likelihood, the latter making the natural-gradient step non-trivial. For count data we use the Poisson NYC Crime task from GS-LVMOGP. Metrics are test NLPD, RMSE, final ELBO, iterations and wall-clock time to reach within a fixed tolerance of a long-run reference ELBO, and the rate of Cholesky failures. Robustness is measured by sweeping $r_\epsilon$ from $10^{-8}$ to $10^{-1}$, mirroring the D-Adaptation claim of insensitivity to its initial guess from $10^{-16}$ to $10^{-2}$.
There are four decisive checks. First, in the conjugate full-batch case, the variational step must approach 1 and recover the closed-form optimum. Second, on Naval, the geometry-matched rule must converge where the Euclidean ablation and Prodigy do not. Third, the lengthscale-normalized rule for $Z$ must show fewer rank-collapse failures than Euclidean DoWG on $Z$. Fourth, as a downstream test, we swap our trainer into the EULBO-SVGP loop on Lunar Lander and Rover. We then check whether best reward per evaluation matches the grid-tuned surrogate with no refit-time tuning.
If the Euclidean ablation matches the full method everywhere, we will conclude that the geometry is unnecessary and report that negative result.
References
- Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models
- Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedules
- DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
- Learning-Rate-Free Learning by D-Adaptation
- Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner
- Approximation-Aware Bayesian Optimization
- Scalable Multi-Output Gaussian Processes with Stochastic Variational Inference