Choosing How Fast an Optimizer Forgets, Online

Adaptive optimizers such as Adam keep running averages of past gradients and past squared gradients, and each running average has a decay rate that sets how quickly old information fades. Recent theory shows these decay rates act as more than smoothing knobs. They are data-weighting schemes. One line of work writes AdaGrad, RMSProp and Adam as a single algorithm whose only difference is the weight placed on each past gradient. Another shows that Adam is exactly an online learner with exponential discounting of past losses, which is why it can follow a target that moves over time. Taken together, these results say that tuning an optimizer's momentum is the same problem as deciding how much to trust old data. On a data stream whose distribution drifts, the right answer depends on how fast the drift is, and that speed is unknown and often changes over time.

We propose a single-objective streaming optimizer that keeps one set of parameters but maintains several gradient-history estimates in parallel, each with a different forgetting rate. Online, it decides which forgetting rate to trust by checking how well each history predicted the gradient that actually arrived next. The aim is to match the best fixed decay rate chosen in hindsight without any tuning, and to beat every fixed rate on streams whose drift speed itself changes. We also test whether the forgetting rate for the gradient direction should be chosen separately from the forgetting rate for the step-size scaling, which standard Adam ties together.

Core Claim

Weighted AdaGrad with Unified Momentum (AdaUSM) supplies the unifying parameterization. It mixes heavy-ball and Nesterov momentum with a weight $a_t$ on each past squared gradient: $a_t=1$ gives AdaGrad, polynomial weights give a provable $\mathcal{O}(\log T/\sqrt{T})$ rate, and exponentially growing weights recover Adam and RMSProp. The authors leave open how to choose $a_t$ from data. Dynamic Regret via Discounted-to-Dynamic Reduction supplies the analysis tool. It proves that a guarantee on discounted regret turns into a guarantee on dynamic regret, meaning regret against a best-in-hindsight parameter that is itself moving. The bound depends on the comparator's path length $P_T=\sum_t\|x^*_{t+1}-x^*_t\|$, the total distance the moving target travels. It also identifies Adam as discounted Follow-the-Regularized-Leader. Our claim joins these two results. The discount factor that balances noise against staleness depends on $P_T$, which is unknown. A learner that hedges over a geometric grid of roughly $\log T$ discount factors can get within a logarithmic factor of the best single discount for every stretch of the stream.

The construction differs from the standard way of hedging over experts. It does not run $K$ full optimizer copies, each with its own parameter vector. It shares one parameter vector $x_t$ and keeps $K$ momentum buffers $m^{(k)}_t=\gamma_k m^{(k)}_{t-1}+(1-\gamma_k)g_t$, where $g_t$ is the stochastic gradient at step $t$ and $\gamma_k$ is the $k$-th discount factor. It also keeps $K$ matching second-moment buffers. Each buffer is scored by its one-step-ahead prediction error $\|m^{(k)}_{t-1}-g_t\|^2$, discounted over time. The buffers are then combined with exponential weights:

$$ w^{(k)}_t \propto \exp\Big(-\eta \sum_{s\le t}\rho^{\,t-s}\,\|m^{(k)}_{s-1}-g_s\|^2\Big), $$

Here $\eta$ is the meta learning rate and $\rho$ is a slow outer discount that lets the selection itself adapt. The update uses $\sum_k w^{(k)}_t m^{(k)}_t$, preconditioned by a second-moment estimate chosen the same way but with its own weights. Choosing the two weightings independently is the decoupling test. The cost is $2K$ extra vectors of size $d$, with no extra forward or backward passes.

This is a data-driven rule for AdaUSM's weight sequence $a_t$, and it is also an adaptive discount for the reduction of Xie et al. It is restricted to a single scalar loss on purpose. For single-objective losses the discounted-to-dynamic theory already exists, so a clean result is reachable.

Research Program

The theory part extends the discounted-to-dynamic reduction to a mixture of discounts. We will first do this for the curved losses where Xie et al. obtain optimal rates, which are linear regression and logistic regression. The target is a bound of the form "best fixed discount on each interval, plus a $\log K$ overhead." The step at risk is that the buffers share one parameter vector, so their prediction errors are not independent expert losses. The analysis must bound the interaction this sharing creates. If that fails, we fall back to the $K$-copy version, where standard expert arguments apply, and report the shared version as a heuristic. The general convex, non-curved case is listed as open by Xie et al. and stays out of scope.

The empirical part runs the optimizer in three regimes. The first is convex streaming problems, where theory and experiment can be compared directly. The second is small neural forecasters and classifiers trained online. The third is a controlled study where the drift speed is set by hand: slow, fast, and switching between the two. A further analysis question is whether the learned weights $w_t$ move toward short memory after a shock and back toward long memory afterward, the behavior FraudFox engineers by hand.

Baselines

The naive reference point is Adam with defaults ($\beta_1=0.9$, $\beta_2=0.999$) and a tuned learning rate. It forgets at a fixed rate that nobody chose for the stream.

The strongest competitor is Adam or AdaUSM whose decay rates are grid-searched in hindsight on each stream. This is an oracle: it sees the whole stream before choosing. We do not expect to beat it on streams with stationary drift speed. Success there means coming within a few percent of its cumulative online loss with no tuning. On switching-speed streams, success means a lower cumulative loss than every fixed setting in the grid. All methods share the learning-rate grid and the same number of tuning runs. The oracle gets its full decay grid on top of that budget, a deliberately generous advantage.

The third baseline is AdaUSM with polynomial weights $a_t=t^\alpha$ from Weighted AdaGrad with Unified Momentum. It has the cleanest stationary theory but never forgets, so it shows what drift costs.

The fourth is a hard sliding window, implemented as periodic optimizer-state resets with a tuned window length. Learning to Forget argues windowing creates boundary artifacts and that a learnable exponential decay is better; that paper learns a decay inside a Gaussian-process forecaster, not inside an optimizer. We include its forecaster as a model-level reference on GluonTS.

Two ablations isolate the key ingredients. The $K$-copy version shows whether sharing one parameter vector costs accuracy. A tied version, which uses one weighting for both moments, shows whether decoupling them matters.

For scale, no reported result found on Lacuna for discounted or adaptive-decay optimizers on these streams. The only directly relevant figure is the ATLAS batch predictor, LGBM-Meta, with Spearman correlation 0.912 on job-runtime prediction (ATLAS). It is an offline ceiling, not an online competitor.

Why not just…?

Why not just grid-search Adam's decay rates? On a stream whose drift speed is stationary, this may work about as well as our method, and we say so. Grid search fails in two cases. First, deployment cannot look into the future. Second, the best rate changes within the stream. The switching-speed experiments and the per-interval regret curves settle whether the second case matters in practice. If our method only ties the oracle everywhere, the contribution shrinks to "tuning-free," and we will report it that way.

Why not just reset the optimizer periodically, or use a sliding window? A window throws away recurring structure. ATLAS reports that 81.3% of test jobs recur from earlier templates (ATLAS). FraudFox's shock experiments show that methods without smooth forgetting stay confused long after a distribution swap (FraudFox). The window baseline with a tuned length is run in every experiment. ATLAS is where we expect it to lose most clearly.

Why not just learn a single decay rate by gradient descent on the decay itself? This is a real threat, and the file contains no optimizer-level version of it to cite. The closest analogue is the variational learning of a decay in Learning to Forget. A learned decay is a single point estimate that has to move after a shock. Hedging keeps the short-memory option available all the time. We will implement hypergradient descent on $\beta_1$ and $\beta_2$ as an additional baseline and compare how quickly each method recovers after an abrupt shift.

Proposed Experiments

The convex track uses online logistic regression on the Covtype stream, as in Multi-Objective Online Learning but with a single loss. We inject controlled drift by rotating the label-generating weights at slow, fast, and switching speeds. This produces cumulative-loss and dynamic-regret curves to compare against the predicted logarithmic overhead. We also reproduce the two-oracle shock setup from FraudFox and measure time-to-recovery after the swap.

The forecasting track trains small neural forecasters online on GluonTS Electricity, Solar, KDDCup and M4, scored by CRPS (continuous ranked probability score). The scheduling track predicts job runtimes on ATLAS in arrival order, scored by Spearman correlation and Cov@25. The vision track repeats the clipped-Adam CIFAR-10 setup of Xie et al. with class proportions shifted over time, to test whether the method survives non-convex training.

The decisive checks are three. On switching-speed streams, does the method beat every fixed decay in the oracle grid? On stationary-speed streams, does it stay within a few percent of the oracle? Does the decoupled weighting choose measurably different memory lengths for direction and scale, and does that difference improve loss compared with the tied ablation? If the first check fails while the second passes, the honest conclusion is "tuning-free forgetting," a smaller but still useful result.

A clean single-objective result would also set up an extension the file flags as missing: tracking a drifting Pareto front with a separate discount for each objective. It is deliberately not attempted here.