Stochastic gradient descent (SGD) can produce more than a point estimate. If you average its iterates (the successive parameter values it visits), the average is approximately Gaussian, and you can build confidence intervals and hypothesis tests in a single pass over a data stream. The usual average is uniform: every iterate gets equal weight, including the early ones produced while the optimizer was still far from the answer. Recency weighting gives later iterates more weight, so the average forgets the starting point faster. That is known to reduce point-estimate error. Nobody has asked whether it also yields more statistical power, meaning the chance that a test detects a real effect at a fixed false-positive rate. Power is what matters when events are scarce, for example in heavily censored time-to-event data, where only a small fraction of patients have an observed outcome.
This note proposes a size-calibrated study of that question. The headline prediction is easy to test. At a fixed number of observed events, the power-maximizing amount of recency weighting is large when events are few and shrinks toward uniform averaging as events accumulate. Weighting that also preserves full asymptotic efficiency, if it can be calibrated at small samples, should dominate both uniform averaging and burn-in. The deliverable is a weighting rule tied to sample size, test critical values matched to that rule, and a measured "events saved at 80% power" relative to existing one-pass procedures.
Core Claim
Two forces set the finite-sample power of a test built from weighted SGD iterates. The first is contamination from the starting point. Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes shows that uniform averaging carries an extra logarithmic penalty in error bounds because early iterates are never forgotten. The second is a variance tax. Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality proves that suffix and polynomial weighting inflate the asymptotic variance by a factor $w > 1$, which widens intervals by $\sqrt{w}$. The same paper shows the Gaussian approximation fails outright if weights are too abrupt or too concentrated.
Lower point-estimate error does not imply more power. A test's power depends on the signal relative to the spread of the test statistic, and that spread includes the noise of whatever estimates the variance. Suppose the variance estimator degrades, or the false-positive rate drifts above nominal. Then a weighting scheme can lower error and still lose power once the test is recalibrated to its nominal size.
We therefore make three claims. The first is that the power-optimal recency exponent is sample-size dependent. The second is that it is measurably nonzero when observed events number in the low hundreds. The third is that the gain is large enough to matter in event-starved regimes, which is the setting SurvHTE-Bench identifies as destabilizing when censoring exceeds 70%.
Research Program
We study a one-parameter family of weights, $w_i \propto i^{\rho}$ with $\rho \ge 0$, where $\rho = 0$ is uniform averaging. We add the adaptive correction from Wei, Zhu and Wu, which approximates the best linear unbiased estimator, as a separate scheme. It is the strongest existing weighting and is designed to reach $w = 1$. The weighted average after $n$ observations is $\tilde\theta_n = \sum_{i \le n} w_{n,i}\theta_i$, where $\theta_i$ is the $i$-th iterate and the weights $w_{n,i}$ sum to one.
For inference we need a statistic that does not require inverting a covariance matrix that is near-singular at small samples. We adapt the random-scaling idea of A Statistical Online Inference Approach in Averaged Stochastic Approximation. That paper builds a test statistic whose limiting distribution is free of unknown nuisance parameters, normalizing by the fluctuation of partial averages along the trajectory. Our weighted analogue is
$$V_n = \frac{1}{n}\sum_{s=1}^{n} \Big(\tfrac{W_s}{W_n}\Big)^2 (\tilde\theta_s - \tilde\theta_n)(\tilde\theta_s - \tilde\theta_n)^\top,$$
where $W_s = \sum_{i\le s} w_i$ is the cumulative weight. The test statistic is $n(\tilde\theta_n - \theta_0)^\top V_n^{-1}(\tilde\theta_n - \theta_0)$ for a null value $\theta_0$.
Its limiting distribution is a functional of Brownian motion that depends on $\rho$. We tabulate critical values for each weight profile by simulation, one table per $\rho$. The research file is internally inconsistent here: it reports that Wei, Zhu and Wu already demonstrate random-scaling intervals for their adaptive weights, and also lists this as an open gap. We assume the former. We reuse their construction where it applies and do not claim the weighted pivot as our contribution.
Our contributions are elsewhere. The first is a power criterion rather than an error criterion for choosing the weights. The second is finite-sample size calibration. The third is evidence in the event-starved survival setting.
The theoretical piece is a finite-$n$ approximation of test power at a local alternative. In this approximation, power depends on the effect size divided by an effective standard error. That standard error combines three terms: $\sqrt{w(\rho)}$, a contamination term that decays at a rate set by $\rho$ and the step-size schedule, and the extra dispersion from self-normalization. Minimizing this approximation over $\rho$ gives a rule $\rho^\star(n_{\text{events}}, \text{step size})$. We fix that rule before running the held-out evaluation settings, so the weights are never tuned on the data being tested.
For survival data we avoid the Cox partial likelihood. Its risk sets make mini-batch gradients biased, which would violate the independence assumptions of every result cited above. Instead we use the piecewise-exponential model. It splits each patient's follow-up into time intervals and fits a Poisson generalized linear model, which is convex and separable per person-interval. This keeps the setting inside the theory of Wei, Zhu and Wu while isolating event starvation as the variable of interest. Gradients carry information only at observed events, so heavy censoring shrinks the effective sample.
Baselines
The ceiling is the offline maximum-likelihood fit with a standard Wald or likelihood-ratio test, run on the identical data. It is not a competitor for streaming use. Its role is to define how much power exists to be recovered. Every one-pass method is reported as a fraction of this ceiling.
The naive reference point is uniform Polyak-Ruppert averaging with random scaling, as in Xie and Zhang. It is the natural default and is reported to give "coverage rates near 95%" in their regression experiments. We run it with the same step-size schedule, the same data order, and the same number of passes (one) as every other method.
The strongest existing weighting is the adaptive best-linear-unbiased weighting of Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality. We run it with both its covariance-based intervals and random scaling. The file reports better finite-sample error for this method on linear, logistic and expectile regression, but gives no figure for that improvement and no power figure. No reported result was found on Lacuna for the power of any SGD-based test.
We include two further competitors for robustness. The plug-in covariance estimator of Statistical inference for model parameters in stochastic gradient descent is reported to achieve "the nominal 95% confidence interval coverage across various dimensions," while its batch-means estimator reaches "90-93% coverage." Implicit SGD with uniform averaging (proxPR) from Statistical inference with implicit SGD: proximal Robbins-Monro vs. Polyak-Ruppert is reported to give "near-nominal coverage (around 95%)."
The ablation that isolates our key ingredient holds everything fixed except $\rho$, sweeping from 0 to aggressive values. It also compares a $\rho$ chosen by our sample-size rule against the best fixed $\rho$ found in hindsight.
All methods share one fairness protocol. Each gets the same step-size grid, tuned once on a separate pilot simulation. Every method's test is size-checked, and any method whose empirical false-positive rate exceeds 6% at nominal 5% is reported with size-adjusted power. Beating a baseline means one of two things: higher size-adjusted power at matched sample size, or reaching 80% power with fewer observed events.
Why not just…?
Why not just discard a burn-in and average uniformly afterward? This is suffix averaging. Wei, Zhu and Wu prove it carries an asymptotic variance tax $w > 1$. Lee, Lee and Won note that choosing the cutoff is fragile: too short leaves contamination, too long throws away scarce data. Still, at small samples the tax may not matter. We therefore include burn-in at three cutoffs as baselines. If the best cutoff matches our rule, the honest conclusion is that a well-chosen burn-in suffices.
Why not just use implicit SGD with a large initial step so early iterates are harmless? This is the most serious threat. Implicit updates tolerate initial steps up to 100 and escape bad starts quickly, which could leave little contamination for weighting to remove. We test this directly by crossing implicit versus explicit updates with $\rho$. If implicit SGD with uniform averaging matches weighted explicit SGD in power, we report that recency weighting is redundant given implicit updates. Gap 5 in the file, combining the two, then becomes the only remaining question.
Why not just refit offline and bootstrap? For a fixed cohort of 50 to 200 patients, refitting offline is cheap and almost certainly best. That is exactly why it serves as our ceiling rather than a competitor. The proposal's scope is settings where data cannot be stored or refit at every interim look, such as continuous monitoring of large health-record streams for rare outcomes. If one-pass uniform averaging already recovers nearly all of the offline power, the premise fails and we will say so. Briscoe et al. (Algorithmic Accountability in Small Data) also show that resampling summaries of very small groups is jagged and unstable, which argues against the bootstrap as a remedy at the very smallest sizes.
Proposed Experiments
The first testbed is the synthetic linear and logistic regression setups used by Chen et al., Xie and Zhang, and Wei, Zhu and Wu. These cover identity, Toeplitz and equicorrelated covariate designs with dimension up to 20. We vary sample size from 100 to 10,000 and set the initialization at near, moderate and far distances from the true parameter.
The second testbed is piecewise-exponential survival streams. These use the synthetic survival configurations of SurvHTE-Bench and its semi-synthetic covariates drawn from ACTG 175 and MIMIC-IV. Censoring is set at 30%, 70% and 90%, and the target is a treatment coefficient, since SurvHTE-Bench reports no scalar results suitable for this purpose.
The metrics are four. Empirical size is measured at nominal 5%. Size-adjusted power curves are traced across effect sizes. Median interval width is recorded. The headline number is the number of observed events needed to reach 80% power, expressed relative to uniform averaging and to the offline ceiling.
Three outcomes would be decisive. If $\rho^\star$ falls toward zero as events grow and its gain concentrates below a few hundred events, the core claim holds. If the adaptive weighting of Wei, Zhu and Wu matches $\rho^\star$ everywhere at nominal size, the contribution narrows: the result becomes evidence and calibration showing that existing adaptive weighting is the power-optimal default. If neither weighting beats implicit SGD with uniform averaging, or neither closes a meaningful fraction of the gap to the offline ceiling, recency weighting does not buy power, and that negative result is worth publishing.
References
- Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
- Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality
- SurvHTE-Bench: A Benchmark for Heterogeneous Treatment Effect Estimation in Survival Analysis
- A Statistical Online Inference Approach in Averaged Stochastic Approximation
- Statistical inference for model parameters in stochastic gradient descent
- Statistical inference with implicit SGD: proximal Robbins-Monro vs. Polyak-Ruppert
- Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics