Modern cloud autoscalers decide how many servers or containers to keep running by forecasting demand and adding a safety margin. The margin is sized from how variable demand appears to be. That variability is computed from monitoring metrics such as request rates and CPU utilization, which are sampled at jittery intervals, aggregated over misaligned windows, and sometimes dropped. A basic fact from measurement statistics applies: zero-mean measurement noise does not average out of a variance estimate. It adds to it. If an autoscaler sees $Y = X + \epsilon$, where $X$ is the true demand and $\epsilon$ is zero-mean noise with variance $\sigma^2$, then the expected sample variance of $Y$ is $\mathrm{Var}(X) + \sigma^2$. Uncertainty-aware autoscalers therefore read monitoring noise as demand volatility and over-provision. Worse, when the noise level shifts, for example because scrape jitter grows with server load, the margin shifts with it for reasons unrelated to demand.
We propose to estimate true demand variance directly by splitting each control interval's raw telemetry into two halves whose measurement errors are independent, then using the cross-covariance between the halves. This borrows the leave-one-out logic used to de-bias treatment-effect covariances. The estimate needs no knowledge of the noise level and remains valid when that level changes over time. We plug it into the predictive variance of a Gaussian-process autoscaler and into the chance constraint of a model-predictive autoscaler. The claim we will test is narrow: the correction matters mainly when the telemetry noise level is non-stationary. In that regime, it beats simply re-tuning the safety margin against observed service-level outcomes.
Core Claim
Our central object is an estimator of demand variance that is unbiased under heteroskedastic, time-varying measurement noise. Within each control interval we hold many raw samples, either several scrapes or per-replica readings. We assign them to two interleaved halves $a$ and $b$ that come from different scrape times or collectors, so their errors are independent of each other and of the true signal. Each half yields a noisy reading of the same underlying demand trajectory. Under those assumptions,
$$\mathrm{Cov}(Y^a_t, Y^b_t) = \mathrm{Var}(X_t),$$
where $Y^a_t, Y^b_t$ are the two half-interval readings at step $t$ and $X_t$ is true demand. The noise term drops out because it is uncorrelated across halves, and its size never enters the formula. This is the same reason the jackknifed covariance estimator in Learning the Covariance of Treatment Effects Across Many Weak Experiments is robust to heteroskedasticity. That paper splits units within each experiment so that error in the predictor is independent of error in the outcome. It reports that its subtract-the-noise estimator cuts median absolute bias by about 63% relative to naive least squares on Netflix experiments. We reuse the splitting idea, but our setting has a temporal stream and a downstream controller, and it has no meta-analysis.
The scope of the claim is deliberately limited. When telemetry noise is small relative to demand variability, or stationary, a well-tuned safety margin already absorbs the bias, and we expect our method to tie. The gain should appear when the noise level changes faster than outcome-based tuning can follow. Under-estimating variance is far costlier than over-estimating it, because it causes violations of the service-level objective (SLO, the latency or availability target the operator promises). For that reason we will report negative-variance rates and clip estimates conservatively. The file flags asymmetric cost as a known blind spot of de-biasing methods.
Research Program
The work has three parts.
Measure how much the noise matters. Before correcting anything, we will establish whether the noise is large enough to matter. The production traces in the file record only what the monitoring system saw, with no ground truth. We will therefore build a Kubernetes testbed that replays the Azure Functions trace from Serverless in the Wild and the Alibaba Cloud traces used by MagicScaler. The testbed keeps exact per-request logs as ground truth and runs a standard Prometheus scrape pipeline as the observed signal. From this we can compute, interval by interval, how much of the measured variance is noise. We can also see whether that share varies with load, scrape interval, and collector contention. If the noise share is negligible under realistic configurations, the project stops there and reports a negative result. That would itself correct an assumption the field has not checked.
Build the split-scrape estimator and integrate it. This includes interleaved splitting rules for counter-based rates and for gauge metrics such as CPU. We will add a shrinkage step toward the raw estimate when the cross-covariance is unreliable, so the estimator is never less conservative than a floor the operator sets. Integration covers two controllers. The first is a Gaussian-process autoscaler in the style of MagicScaler, where the de-biased variance replaces the empirical noise and kernel variance terms. The second is a chance-constrained model-predictive controller in the style of OptScaler. There, the split estimate also feeds the reactive linear CPU estimator, so that high-frequency noise is not read as a shift in load structure.
Analyse closed-loop behaviour. We will characterise how the correction interacts with scaling latency and with instance thrashing. In particular, we will check whether removing inflated variance lets the controller run with less smoothing, and so with less lag, without more oscillation. Smoothing lag is the trade-off that Singhvi et al. and the threshold literature identify as the core failure of simple rules.
Baselines
The naive reference point is the same Gaussian-process and model-predictive controllers fed raw telemetry variance, with their safety parameters tuned once on a validation week. This is how the systems in the file operate, and it isolates the effect of noise inflation.
The strongest existing uncertainty-aware method is MagicScaler. MagicScaler couples a multi-scale attention feature extractor with Gaussian-process regression and plans over a 12-step horizon. On Alibaba Cloud traces it reports reducing QoS violations by "nearly 90%" for "only a marginal 1.2% increase" in resource cost. No public code is listed in the file, so we will reimplement it. We will report our reimplementation's numbers against its reported ones rather than claim parity. OptScaler is the second strong competitor because its chance constraint consumes variance directly. It reports "over a 36% reduction" in SLO violations on the Azure Functions workload sets. RobustScaler, which provisions $\kappa$ arrivals ahead of a non-homogeneous Poisson forecast, will also be run; the file reports no raw hit-rate scalar for it. All methods get the same replayed traces, the same testbed, and the same validation budget for hyperparameter search, measured as the number of tuning trials.
The baseline that most directly threatens our claim is outcome-calibrated margin tuning. It keeps raw variance but continually re-tunes the safety quantile, or RobustScaler's $\kappa$, so that observed SLO violations match the target. A second baseline is static noise subtraction. It estimates $\sigma^2$ once offline and subtracts it, as in the total-covariance estimator, so it is robust only if noise is stationary. The key ablation replaces our per-interval split with a single global split-scrape noise estimate. This separates the value of the splitting trick from the value of tracking heteroskedasticity.
Beating the baselines means sitting on a better Pareto frontier of SLO-violation rate against instance-hours. It also means recovering to target violation rates in fewer control intervals after a noise-regime shift. Under stationary noise, we should match outcome-calibrated tuning with a fraction of its tuning data.
Why not just…?
Why not just smooth the metrics with a moving average and keep threshold rules? Smoothing reduces noise in the mean, but it delays genuine bursts, and it does not remove the additive variance bias for any controller that sizes margins from variance. Yin et al. (An Adaptive Elastic Multi-model Big Data Analysis and Information Extraction System) report that traditional threshold methods cut cumulative resource occupancy by only 30.4%, against 74.4% for their adaptive scheme. We will include an EMA-plus-threshold controller tuned over smoothing constants, and plot its lag-versus-thrashing curve against ours.
Why not just tune the safety margin against observed SLO outcomes? This is the real threat. SLO violations are measured on actual requests, not on noisy telemetry. Calibrating $\kappa$ or the quantile on them therefore absorbs a constant variance inflation automatically, and under stationary noise it may fully match our method. Its weakness is speed. SLO violations are rare by design, so the calibrator needs many intervals to detect that the noise level has moved. Our estimator sees the change within one interval from the telemetry itself. The noise-shift experiment below is built to settle this. If outcome calibration recovers as fast as we do, our contribution shrinks to a diagnostic result.
Why not just use sign-based dynamics that ignore magnitude noise? Robust-to-Noise Algorithms for Distributed Resource Allocation and Scheduling shows sign-based updates tolerate additive link noise. The authors themselves document persistent chattering near the optimum and rely on a heuristic stopping rule. In capacity management, chattering means repeated launches and drains. That work also addresses consensus communication noise rather than variance-based margin sizing. We will count scale actions per hour as the thrashing metric for any sign-filtered variant.
Why not just train a bigger forecaster and let the loss absorb the noise? Fitting noisy targets does not cancel the noise inside a variance estimate. The file's vision results (Noise causes slant underestimation in stereo and motion, Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image) show optimizers systematically fitting zero-mean noise unless the objective is explicitly corrected. Our correction is also orthogonal to forecaster choice, so we apply it to the strongest forecaster we can run.
Proposed Experiments
The first experiment is the noise audit. We replay Azure Functions and Alibaba Cloud traces with ground-truth request logs and Prometheus at scrape intervals from 5 to 60 seconds. We report the fraction of measured variance attributable to noise and its correlation with load. For the estimator itself, we report bias and negative-estimate rate against ground truth for raw, static-subtraction, global split, and per-interval split variance. These results are compared against the 63% bias reduction the covariance paper reports for its estimator in its own domain.
The second experiment is closed-loop evaluation on the testbed under stationary noise. It covers all baselines, and we report SLO-violation rate, instance-hours, cold starts, and scale actions per hour. The Alipay Applets workloads are included where OptScaler's published characterisation allows faithful synthetic replay.
The decisive experiment is a noise-regime shift. Midway through a replay we double scrape jitter or co-locate a contending collector, leaving true demand unchanged. We then measure how many intervals each method needs to return to its target violation rate and how many excess instance-hours it spends in the meantime. If split-scrape variance recovers substantially faster than outcome-calibrated tuning, at equal or lower cost, the claim holds. If the two curves overlap, we will report that the noise problem is real but already solved by margin calibration.
References
- Learning the Covariance of Treatment Effects Across Many Weak Experiments
- Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
- MagicScaler: Uncertainty-aware, Predictive Autoscaling
- OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud
- SNF: serverless network functions
- An Adaptive Elastic Multi-model Big Data Analysis and Information Extraction System
- Robust-to-Noise Algorithms for Distributed Resource Allocation and Scheduling
- Noise causes slant underestimation in stereo and motion
- Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image