Many forecasting models take two kinds of input. The first is the target series' own history, for example past electricity load. The second is outside information that drives the target without being part of it, such as temperature, holidays or a planned price change; these are called exogenous variables. The parts of the network that read exogenous inputs see something unusual. Most of the time those inputs barely matter. Occasionally a heat wave or a holiday arrives and they matter enormously.
We argue that this rare-event structure explains why second-order optimizers are so hard to tune on these models. Second-order optimizers rescale each gradient by an estimate of the loss surface's curvature, and the most familiar example is Shampoo, which builds that estimate from small per-layer matrices. These optimizers keep a running average of curvature that forgets rare-event directions between events. When an event recurs, the curvature estimate is near zero in exactly the directions the event activates. The step size there is then set almost entirely by the damping constant, a small number added before inverting the curvature. Damping is therefore effectively a learning rate for the rare-event pathway, and its best value depends on how rare the events are.
We propose to measure this effect and then fix it with an optimizer that reads the exogenous inputs themselves. When a minibatch contains an exogenous shock, which can be detected from the inputs without labels, the optimizer stores that batch's curvature directions in a slowly decaying memory. That memory sets a floor under the preconditioner, the matrix that rescales the gradient, for the exogenous-pathway layers only.
Core Claim
The claim has three parts, and each can fail independently. First, in forecasters with a separate exogenous pathway, the sensitivity of Kronecker-factored optimizers to damping is concentrated in the exogenous-pathway parameter blocks. Kronecker-factored means the curvature of each weight matrix is approximated by two smaller matrices. Sweeping damping on those blocks alone should reproduce most of the whole-model sensitivity. Second, the mechanism is that the averaged curvature forgets rare-event directions. If so, the usable damping range should narrow as the synthetic events get rarer, and training-loss spikes should coincide with event batches arriving in nearly null curvature directions. Third, an event-triggered curvature memory should widen the usable damping range and lower error inside event windows. It should do this beyond what three simpler fixes achieve: per-block damping normalization, a longer averaging horizon, or oversampling event windows.
For an exogenous-pathway block $b$, the proposed preconditioner is
$$\hat C_b = C_b^{\mathrm{EMA}} + \rho\, U_b \Lambda_b U_b^{\top} + \epsilon I,$$
where $C_b^{\mathrm{EMA}}$ is the optimizer's usual running-average Kronecker factor. $U_b \Lambda_b$ are the top-$k$ eigenvectors and eigenvalues of curvature accumulated only on batches flagged as exogenous shocks, decayed slowly. $\rho$ is the weight on that memory and $\epsilon$ is the ordinary damping. The flag is computed from the exogenous inputs. A batch is flagged when some exogenous variate's standardized deviation from its rolling statistics crosses a threshold. This is the point where exogenous variable integration enters the optimizer itself rather than only the architecture.
Relation to Prior Work
Several papers motivate the gap. On the exogenous side, TimeXer compresses each exogenous series into one token. A single learned token then queries those exogenous tokens through cross-attention, a layer that lets one set of tokens read from another. The authors report that the model underestimates sudden traffic peaks. ChronosX attaches small injection blocks for covariates to frozen pretrained forecasters, and its synthetic suite with spikes, steps and bells gives us events with known ground truth. Overcoming the Modality Gap in Context-Aided Forecasting shows that models learn to ignore auxiliary context when it is noisy. All three train with first-order optimizers and never ask how the exogenous pathway shapes curvature. We keep their architectures fixed and change only the optimizer.
Other papers supply the method. Shampoo introduced Kronecker-factored preconditioning, which is known to be sensitive to tuning and to rely on grafting onto Adam. KL-Shampoo re-derives the factor estimates as a covariance-fitting problem and removes the need for grafting, but it was tested only on language models. Hessian Backpropagation gives modular block-diagonal curvature, which we use to measure per-block spectra exactly on small models. It was evaluated only on CIFAR-scale vision.
Another group constrains the design. Second-Order Optimization with Lazy Hessians shows that stale curvature must be compensated by regularization that scales with how fast curvature changes. Our event memory is an input-triggered, block-local version of that compensation. It differs from the paper's monolithic dense-Hessian analysis. Second-order Optimization under Heavy-Tailed Noise warns that heavy-tailed gradients force clipping thresholds, which are yet another sensitive hyperparameter. We therefore treat clipping as a competitor rather than a free add-on. Bernacchia et al. and the improved empirical Fisher work both note that no principled damping rule exists across architectures. We do not claim to supply one in general, only for the specific rare-event structure.
Finally, Preconditioners for the Stochastic Training of Neural Fields shows that dense-curvature models gain from curvature-aware preconditioning over Adam. That result motivates trying second-order methods here at all, but it says nothing about rare inputs.
Research Program
The work has three stages. The first is measurement. On small TimeXer and ChronosX-adapter models, we compute exact block-diagonal curvature spectra with Hessian Backpropagation at checkpoints. We track the eigenvalues of exogenous-pathway blocks, including the cross-attention projections in TimeXer and the injection blocks in ChronosX. We record how those eigenvalues evolve between and during event batches, and compare them with the optimizer's running-average estimate.
The second stage is the sensitivity study. For SOAP-style and KL-Shampoo optimizers, we sweep damping over five orders of magnitude in two ways: globally, and only on exogenous blocks with the other blocks held at their best value. We summarize each sweep by its sensitivity width, the range of $\log_{10}\epsilon$ over which final validation loss stays within 5% of the best.
The third stage implements the event memory on top of KL-Shampoo, adding the three hyperparameters $k$, $\rho$ and the flag threshold. We report how sensitive the method is to those new knobs as well. A method that trades one fragile hyperparameter for three would not be a win.
Baselines
The naive reference point depends on the dataset. For the foundation-model setting it is the base model run zero-shot without covariates, which is Chronos for ChronosX experiments. On the air-quality and traffic data it is a Seasonal Naive forecast, which Air Quality Arena found hard to beat on the volatile India networks. For context, ChronosX reports a 22% improvement in weighted quantile loss and mean absolute scaled error over zero-shot Chronos on its 32 synthetic datasets (ChronosX). ST-ResNet reports a root mean square error (RMSE) of 16.89 on TaxiBJ and 6.33 on BikeNYC. We found no reported per-benchmark numbers on Lacuna for TimeXer, so we will report our own reproductions with the official thuml/TimeXer code.
The strongest practical competitor is AdamW with separate, independently tuned learning rates for the endogenous and exogenous branches. It is cheap and already gives the exogenous pathway its own step scale, so it addresses the same symptom without curvature. It gets the same tuning budget as every other method: 30 trials of the same hyperparameter-search algorithm per dataset, three seeds, and equal wall-clock time.
The strongest second-order competitor is KL-Shampoo with one global damping value, tuned within that same budget. It is also run with trace-normalized per-block damping, which sets each block's damping in proportion to its mean curvature eigenvalue. That heuristic is the obvious way to make damping scale-free per block.
Three ablations isolate the key ingredient. The first uses a longer averaging horizon for exogenous-block factors only, with no event detection. The second keeps the memory but flags batches at random at the same rate, which tests whether reading the exogenous inputs matters. The third applies the memory to all blocks, which tests whether restricting it to exogenous blocks matters.
The proposal beats a baseline only if it meets two conditions on at least two of the three model families. Its sensitivity width must be at least one order of magnitude wider. Its event-window mean squared error must be lower, by more than the across-seed standard deviation. Both comparisons are made with average error held equal or better.
Why not just…?
Why not just give the exogenous branch its own Adam learning rate? This is a real threat. For small forecasters, curvature structure may matter less than step scale. The neural-fields evidence in Preconditioners for the Stochastic Training of Neural Fields favors curvature-aware methods only when curvature is dense, and we do not yet know whether exogenous cross-attention blocks are dense. The measurement stage answers that question directly. If per-branch AdamW matches our method on both sensitivity width and event error, we will report that second-order methods are unnecessary here. In that case the contribution narrows to the diagnosis.
Why not just normalize damping per block by its curvature trace? Trace normalization corrects for blocks having different overall scales. It does not correct for a block whose curvature is nearly zero in some directions between events, because the trace is dominated by the directions that are always active. The comparison with trace-normalized KL-Shampoo tests this. If it closes the gap, our mechanism claim is wrong.
Why not just oversample event windows or use a longer averaging horizon? Oversampling attacks forgetting from the data side, and it is the alternative we consider most likely to work as well. It also requires knowing in advance which windows are events, which is the same signal our flag uses. We therefore run event oversampling at matched effective frequency as a full competitor. The longer-horizon ablation tests whether plain memory, without event selection, is enough. Our method is justified only if it beats both. A plausible reason it might is that oversampling also changes what the model fits, while our memory changes only how the model steps.
Why not just clip gradients? Clipping limits the explosion when an event batch arrives in a nearly null direction. But it adds a threshold that is itself sensitive, as argued in Second-order Optimization under Heavy-Tailed Noise, and it suppresses exactly the gradients that carry event information. We include tuned clipping on top of global-damping KL-Shampoo and measure its event-window error, not just its stability.
Proposed Experiments
The mechanism test runs on the ChronosX synthetic generator, where we control spike frequency over 1%, 5% and 20% of windows. The prediction is that sensitivity width for global damping shrinks as events get rarer, while width for the event-memory optimizer stays roughly flat. We also predict that more than half of training-loss spikes fall on flagged batches.
The real-data tests cover four settings. They are ChronosX adapters on Chronos for M5 and ETT, TimeXer on the electricity price (EPF) datasets and Traffic, and an ST-ResNet reproduction on TaxiBJ and BikeNYC with its weather and holiday stream. The AQA-Bench 168-hour-to-24-hour task tests whether meteorological covariates help with the volatile CPCB series.
To probe modality collapse, we train a DoubleCast-style model (ServiceNow/DoubleCast) on CAF-7M. The question is whether damping controls whether the context pathway ever activates. The reference point is the reported 71% win rate of DoubleCast over Chronos on the verified test set (Overcoming the Modality Gap).
Metrics are RMSE and MASE overall, MSE restricted to event windows, sensitivity width, and wall-clock time to target loss. The decisive check is the exogenous-only damping sweep. If sweeping damping on exogenous blocks reproduces less than half of the whole-model sensitivity, the localization claim fails, and the event memory has no reason to exist.
References
- TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables
- ChronosX: Adapting Pretrained Time Series Models with Exogenous Variables
- Overcoming the Modality Gap in Context-Aided Forecasting
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Understanding and improving Shampoo and SOAP via Kullback-Leibler Minimization
- Modular Block-diagonal Curvature Approximations for Feedforward Architectures
- Second-Order Optimization with Lazy Hessians
- Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
- Exact natural gradient in deep linear networks and its application to the nonlinear case
- An Improved Empirical Fisher Approximation for Natural Gradient Descent
- Preconditioners for the Stochastic Training of Neural Fields
- Air Quality Arena: A Large-Scale Multi-Region Ground Monitoring Dataset and Benchmark for Air Quality Forecasting with Time-Series Foundation Models
- Predicting citywide crowd flows using deep spatio-temporal residual networks