Systems that tune themselves while running face two problems. A storage engine, a database, or a serving cluster can often run faster by changing its settings. But every change has a price: data gets reorganized, memory is reallocated, or model weights are reloaded. A tuner that changes settings too eagerly "thrashes," spending more on transitions than it gains. The usual model behind such tuners is a Gaussian process (GP), a regression model that also reports how uncertain each prediction is. A GP becomes too slow to update once it has seen tens of thousands of telemetry points, because exact inference costs grow with the cube of the number of points. The work that makes switching costs explicit in Bayesian optimization (BO, the search procedure that uses the GP to pick the next setting) runs on exact GPs and on problems of two to eight dimensions. The work that makes GPs scale assumes that moving to any new setting is free.
This note proposes a compressed GP whose limited memory is spent only where a "should we switch now?" decision is being made. The compressed model summarizes all data through a small set of $m$ representative "inducing points." We train those points, and the model itself, against a single quantity: the expected payback of a switch. Payback means the predicted gain, accumulated over how long the new setting is expected to stay useful, minus the one-time cost of the transition. The same quantity then gates whether the system actually reconfigures.
The less obvious claim is about safety. Focusing a compressed GP on promising regions is known to make it blind to distant optima. Under switching costs this blindness should become nearly harmless, because distant moves are exactly the ones the transition cost already discourages. If that claim holds, the cost structure that makes online tuning hard also makes aggressive model compression safe.
Core Claim
The proposal makes two linked claims. The first is a method claim. Take a sparse variational GP over the joint space of configuration and workload descriptor. Train it with a lower bound on expected net payback rather than on global data fit. Run it inside a streaming update loop with fixed cost per step. This should achieve higher cumulative net performance (gains minus all transition costs) than the switching-cost acquisition rules in the file paired with a conventionally trained sparse GP. The advantage should hold at matched inducing budget and matched per-decision wall-clock.
The second is an analytical claim. The known bias of utility-focused inducing points shrinks as switching cost grows relative to per-step gain. A pure-exploration setting therefore overstates how risky this kind of approximation is.
Relation to Prior Work
Three papers motivate the problem by showing that switching costs must enter the acquisition rule. An adaptive approach to Bayesian Optimization with switching costs proposes EIPU, which divides expected improvement by cost and compares a "keep the costly setup" candidate against a "switch" candidate. It adds cost cooling, which shrinks the penalty as the evaluation budget runs out. Bayesian optimization for modular black-box systems with switching costs (LaMBO) locks expensive pipeline stages using a tree-structured, slowly moving bandit. Movement Penalized Bayesian Optimization with Application to Wind Energy Systems (GP-MD) prevents erratic jumps by running mirror descent over a hierarchically separated tree of discrete actions.
All three use exact GPs, and the two tree methods need discrete action spaces. EIPU's cost cooling is tied to a fixed evaluation budget, which does not exist in a system that runs indefinitely. We keep their core insight that switching must be priced. We replace budget-based annealing with a horizon set by how long workload regimes last, and we replace exact inference with a compressed model trained on the decision.
The method is supplied by Approximation-Aware Bayesian Optimization. Its EULBO objective trains inducing points and the acquisition jointly so that model capacity lands near suspected optima. The paper also flags the cost: the model can miss peaks outside its focus region. We reuse the EULBO construction but swap expected improvement for net payback. We then test whether switching costs neutralize its stated failure mode.
For streaming, Streaming Sparse Gaussian Process Approximations recycles the previous approximate posterior as the prior for the next batch. Recurrent Memory for Online Interdomain Gaussian Processes updates kernel quantities by ordinary differential equations and reports 0.4 seconds versus 140 seconds for OSGPR, an online sparse GP that re-optimizes its inducing points. That gap is evidence that re-optimizing the surrogate's own representation is a real overhead. We use the Bui et al. recursion for cheap updates at every step and rerun the payback-trained inducing optimization only occasionally.
The systems papers constrain the design and supply evaluation settings. Eva: Cost-Efficient Cloud-Based Cluster Scheduling and HexGen-3 both reconfigure only when predicted savings over the new configuration's lifetime repay the migration cost. They use deterministic predictions, and HexGen-3 assumes deterministic hardware latency. Our payback quantity is their rule made probabilistic and connected to the model's own uncertainty. CAMAL phases structural changes to the storage engine into background compaction, which gives a realistic, state-dependent switching cost. MLOS documents that many knob changes require restarts. Optimizing Decision Quality with Contract Algorithms argues that the time an optimizer spends deciding can cancel the benefit of its decision. That argument is why per-decision latency is a primary metric here.
Research Program
The model is a sparse variational GP $q(f)$ over pairs $(x, c)$, where $x$ is a configuration and $c$ is a workload descriptor such as read/write mix or request length distribution. The system currently runs configuration $x_t$ under context $c_t$, and moving to candidate $x'$ costs $C(x_t, x')$. The net payback of switching is
$$ U(x' \mid x_t, c_t) = \big(f(x', c_t) - f(x_t, c_t)\big)\, H(c_t) - C(x_t, x'), $$
where $f$ is the performance objective and $H(c_t)$ is the expected time the current workload regime will persist. We estimate $H$ from the empirical durations of past regime changes in $c$. We deliberately do not propose a new change detector; RSOBSO-style (An Online Non-Stationary Simulation Optimization Approach Based on Regime Switching) regime models can be dropped in.
The cost $C$ is either known (restart time, compaction volume) or itself learned by a second small GP over measured transition durations. Following EULBO, the inducing locations $Z$, the variational parameters, and the candidate $x'$ are optimized jointly by maximizing
$$ \mathcal{L}(q, Z, x') = \log \mathbb{E}_{q}\big[\max(U(x' \mid x_t, c_t), 0)\big] + \mathrm{ELBO}(q, Z), $$
where ELBO is the standard data-fit bound. The first term pulls inducing points toward the incumbent and toward candidates whose payback is plausibly positive.
The throttle is simple. The system switches only when a lower credible quantile of $U$ at the chosen $x'$ is positive; otherwise it stays and keeps collecting telemetry at $x_t$. Between full re-optimizations, new observations enter through the closed-form streaming update with $Z$ held fixed. A full re-optimization is triggered when the sign of the payback's credible interval changes, or when $c$ enters a new regime. This connects the surrogate's internal "reconfiguration" schedule to the same payback signal that gates the system's own reconfiguration.
The analytical component bounds the extra regret caused by excluding regions from the inducing set. Under a Lipschitz cost $C(x, x') \ge \lambda \lVert x - x' \rVert$, any candidate at distance $d$ yields positive payback only if its predicted gain exceeds $\lambda d / H$. This rules out a growing radius of candidates as $\lambda / H$ rises. The bias from leaving those regions unmodeled should therefore fall roughly in proportion to the fraction of the space that can still pay back. We will check this relationship empirically before trying to prove it.
Baselines
The naive reference points are two. The first is a static configuration found by offline tuning, which MLOS shows leaves 20% to 90% of throughput on the table in SQL Server components. The second is unthrottled GP-UCB on the same compressed GP, which the LaMBO neuroimaging study shows thrashing: it needed 5.6 hours to reach 95% optimality against LaMBO's 1.4 hours.
The strongest competitor on the core question is EIPU with cost cooling from Pricopie et al. It is the most directly comparable principled switching-cost rule and it needs no tree structure. Because exact EIPU cannot run at our data scale, we pair it with a standard ELBO-trained sparse GP that uses the same $m$, the same kernel, and the same streaming recursion. Cost cooling is driven by an equivalent horizon in place of the evaluation budget. On small instances, where an exact GP is feasible, we also run exact EIPU to bound how much the approximation itself costs. The file gives no numeric result for EIPU, only that it outperformed or matched heuristics at 16x and 32x switching-cost ratios; no reported number was found on Lacuna for EIPU's GAP value. Beating it means higher cumulative net performance at equal $m$ and equal wall-clock, with non-overlapping intervals across seeds.
On discrete-action tasks the strongest competitor is GP-MD, run on its own wind-energy setting. The file reports only that GP-MD achieved higher net energy than contextual GP-LCB; no reported number was found on Lacuna for its net energy value. Beating it means more net energy over the same 960 hours.
On RocksDB we compare against CAMAL. It reports a 28% average latency improvement on its YCSB workloads and up to an 8x write-latency reduction on write-heavy ones. We run CAMAL's released configuration on the same 24 shifting test workloads with the same sampling-time allowance. Beating it means lower total latency, with transition-induced latency counted.
Two ablations isolate the key ingredients. The first keeps our payback throttle and streaming loop but trains inducing points by ELBO alone; this separates the decision-trained representation from the throttle rule. The second keeps everything but fixes the horizon $H$ to a constant, which isolates the contribution of regime-aware amortization. All methods share hyperparameter-tuning trials and per-step compute caps, measured on the same hardware.
Why not just…?
Why not just use EIPU with an off-the-shelf sparse GP? This is the real threat, and it is exactly the first ablation. In low dimensions with generous $m$, an ELBO-trained sparse GP may capture the incumbent-versus-candidate gap well enough that the trained representation adds nothing. Maus et al. found the advantage of decision-trained inducing points grows with dimension and tight budgets; they report about 10 times fewer function evaluations on high-dimensional control. We therefore sweep $m$ and dimension. If ELBO-trained surrogates match ours across the sweep, the contribution reduces to the horizon-based throttle, and we will report it that way.
Why not just add a cooldown timer or a hysteresis threshold? Pricopie et al. show that fixed reuse probability and periodic switching fall into a tuning trap: they are very sensitive to their one parameter and lose to EIPU at high switching costs. We include a cooldown timer tuned with the same trial budget as our method. We evaluate it on workload sequences whose regime durations differ from those it was tuned on, which is where a fixed timer should break.
Why not just apply Eva's rule (savings over expected lifetime exceed migration cost) with a point-estimate regressor? This is our throttle without uncertainty, so we run it as a baseline using the posterior mean only. The prediction is that it over-switches early, when the model is wrong but confident in its mean, and under hardware noise of the kind HexGen-3 ignores. If it matches us on stable traces, the benefit of uncertainty is confined to noisy or early regimes, and we will say so.
Why not just use a sliding-window exact GP or local experts? Sliding windows forget. Chen et al. show sharp spikes in predictive error on recurring tasks, and the ATLAS Alibaba trace reports 81.3% of jobs are recurring (ATLAS), so recurrence is the normal case. Local experts such as SkyGP (Streaming Generated Gaussian Process Experts) are very fast: 0.04 ms per update on SARCOS. We include SkyGP-Fast as an alternative surrogate under our throttle. That comparison tests whether speed alone, without decision-focused representation, suffices.
Proposed Experiments
The first testbed is RocksDB under CAMAL's shifting YCSB workloads, from 10 GB to 100 GB. Switching the level size ratio incurs measured compaction cost, and the context is the workload mix.
The second is Pricopie et al.'s synthetic switching suite (Ackley, Rosenbrock, Schwefel, cost ratios 1x to 32x). We extend it to 20 to 60 dimensions and 10^4 to 10^5 steps with a drifting context. The extended suite stresses scaling where exact methods fail and lets us vary the switching-cost scale $\lambda$ continuously.
The third is the airborne wind energy task: 25 altitudes, 960 hours of European wind traces. It tests transfer to a physical movement cost, against GP-MD.
As a stretch goal, we would build a simulator from the HexGen-3 serving traces (WildGPT to OpenR1-Math shifts), using reload time as the switching cost.
Metrics are cumulative net performance including transition costs, number of switches, regret against a hindsight-optimal switching schedule on synthetic tasks, and per-decision wall-clock.
The decisive check is a sweep of $\lambda$ on the synthetic suite with a deliberately placed distant optimum. At $\lambda = 0$, our method should show EULBO's known blind spot and lose to the ELBO-trained ablation. As $\lambda / H$ grows, the gap should close and then reverse. If the reversal never appears, the analytical claim is false, and the proposal reduces to a horizon-aware throttle.
A second check measures whether payback-triggered re-optimization of the inducing points matches always-reoptimize quality, at a small fraction of OSGPR-style cost.
References
- An adaptive approach to Bayesian Optimization with switching costs
- Bayesian optimization for modular black-box systems with switching costs
- Movement Penalized Bayesian Optimization with Application to Wind Energy Systems
- Approximation-Aware Bayesian Optimization
- Streaming Sparse Gaussian Process Approximations
- Recurrent Memory for Online Interdomain Gaussian Processes
- Eva: Cost-Efficient Cloud-Based Cluster Scheduling
- HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling
- CAMAL: Optimizing LSM-trees via Active Learning
- MLOS: An Infrastructure for Automated Software Performance Engineering
- Optimizing Decision Quality with Contract Algorithms
- An Online Non-Stationary Simulation Optimization Approach Based on Regime Switching
- ATLAS: Alibaba Dataset and Benchmark for Learning-Augmented Scheduling
- Streaming Generated Gaussian Process Experts for Online Learning and Control: Extended Version