When to Stop Trusting the Solar Forecast: Drift-Aware Handoff in Stochastic Control

A battery controller in a solar-powered home or microgrid plans by looking ahead. At each step it asks a forecaster for the next day of sunshine, solves for a charging schedule, applies the first action, and repeats. Many good solar forecasters are autoregressive: they predict one step and feed that prediction back in as input for the next step. Their errors therefore compound, and by the end of the window the forecast has often drifted or flattened into a smooth average. The controller cannot tell any of this. It treats a confident-looking but meaningless 20-hour-ahead number exactly like a reliable 30-minute-ahead number.

We propose a different arrangement. The controller uses the forecaster only up to a cutoff step, chosen anew at every decision from the current weather, at which point the forecast stops being more informative than plain climatology (the historical distribution of sunshine for that time of day and season). Beyond that cutoff, the controller relies on a value function, meaning a learned estimate of future cost as a function of battery and thermal state, computed offline from climatological scenarios by stochastic dual dynamic programming. The claim is twofold. First, closed-loop cost depends on where the forecast's skill runs out, not on forecast accuracy averaged over the window. Second, that point moves a great deal between clear and cloudy days, so a cutoff chosen per decision should beat any fixed planning horizon, however well tuned.

Core Claim

The two pieces this proposal joins already exist separately. On a domestic microgrid with solar panels, a battery, a hot-water tank and a building thermal model, Stochastic optimal control of a domestic microgrid equipped with solar panel and battery and its MPC/SDDP companion compare two controllers. The first is deterministic model predictive control (MPC), which re-plans every step using a point forecast. The second is stochastic dual dynamic programming (SDDP), which builds an offline value function from scenario distributions. SDDP was nearly 45% cheaper in summer and won in 93% of summer scenarios. These papers motivate our proposal, because they show that a value function trained on the uncertainty distribution beats a forecast-driven planner. They also leave a gap. SDDP ignores the information that a good short-term forecast carries, and MPC trusts the forecast at every lead time. Neither paper models how the forecaster's errors grow with lead time.

On the forecasting side, ProbTS shows that autoregressive forecasters accumulate error across 96-step horizons, and that solar data in particular is strongly non-Gaussian. WEATHER-5K finds that data-driven models lose skill sharply with lead time and miss extremes. On the 99.5th-percentile extreme-event score, the best data-driven model reached 14.1% against 37.4% for a physics-based weather model. Both papers measure only open-loop forecast error, and neither looks at what a controller does with the forecast.

The theoretical anchor is Distributed Truncated Predictive Control. For linear time-invariant systems, it proves that the sensitivity of control regret to a forecast error decays exponentially with that error's lead time. It assumes the forecast errors at different lead times are independent and additive. An autoregressive forecaster violates that assumption, because its error at lead $k$ is a compounded function of its earlier errors. Distant errors are therefore both larger and correlated with near-term ones. Our hypothesis is that this growth, not the regret-sensitivity decay alone, sets the useful horizon, and that it varies with the weather.

Research Program

The controller is a hybrid. At decision time $t$, it plans over steps $1,\dots,k^*(t)$ using scenarios from the forecaster. It then uses the SDDP value function $V$, trained offline on climatological scenarios, as the terminal cost at step $k^*(t)$. The cutoff is

$$k^*(t) = \min\{k : \hat e_t(k) \ge \gamma\, \sigma_{\mathrm{clim}}(t+k)\},$$

where the three quantities are defined as follows. $\hat e_t(k)$ is the predicted root-mean-square error of the forecast at lead $k$, given the current weather state. $\sigma_{\mathrm{clim}}(t+k)$ is the climatological standard deviation of irradiance at the target time. $\gamma$ is a single scalar near one, tuned once on validation closed-loop cost. In words, the forecast is handed off to $V$ once its expected error reaches the spread of climatology, because from that point $V$ already encodes everything the forecast still knows.

Estimating $\hat e_t(k)$ honestly is the main technical difficulty. Ensemble spread is the obvious estimator, but Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated? finds that forecasters grow increasingly overconfident with lead time. Transformer Conformal Prediction for Time Series likewise reports that coverage falls below target at long horizons. We therefore do not trust raw spread. Instead, we fit a small regression on held-out rollouts. It maps lead time and cheap regime features to realized error. The features are the ensemble spread, the recent rate of irradiance change, and the satellite-derived "easy" versus "hard" cloud category used by SolarCube. ARROW's finding that rollout error depends on the regime, motivating fine steps during transitions and coarse steps when conditions are stable, supplies the intuition. We reuse that idea but apply it to the control horizon rather than the forecast step size.

Inside the window, scenarios are built in a way that respects compounding. We bootstrap whole error paths from the forecaster's past rollouts, which preserves the correlation between lead times, instead of sampling an independent error at each lead. This tests the independence assumption in Distributed Truncated Predictive Control directly. We also exploit the non-negativity of solar irradiance. Following the ensemble-deduction idea in Modified Auto Regressive Technique, errors are added to residuals from the expected daily solar curve and then clipped at night. This avoids the non-physical negative generation that unconstrained perturbations would produce.

The analytic component is deliberately modest. For a linear storage model with convex cost, we will decompose the gap between the hybrid controller and an oracle controller into three parts: a forecast-window term, a terminal value-function error term, and a mismatch term at the handoff. The aim is to show that the first term grows with $k$ under compounding errors while the second does not. That would justify a crossover rule of the form above. Value-function error is treated as in Error Propagation in Dynamic Programming, which separates regression error, Monte Carlo error and error propagated backward through the horizon.

Testbed

Forecasters are trained on SolarCube. It provides 15-minute ground-station irradiance at 19 sites, matched with satellite imagery, and labels easy and hard cloud scenarios. A 24-hour window at this resolution is 96 steps, the same scale at which ProbTS documents error accumulation. We compare three forecasters. The first is an autoregressive transformer with RevIN, a reversible normalization that removes each input window's level and scale before forecasting and restores it afterward. The second is an ensemble-deduction autoregressive model in the style of the Modified Auto Regressive Technique. The third is a non-autoregressive direct forecaster in the style of PatchTST, which predicts all 96 steps in one pass.

The station irradiance then drives a reimplementation of the domestic microgrid from the Météo-France/SDDP papers: battery, hot-water tank, an R6C2 thermal model (a lumped resistor-capacitor model of the building) with four state variables, and StRoBe occupant demand profiles. The file gives no public resource for that simulator, so it will be rebuilt from the paper's description. We will first check that our MPC-versus-SDDP gap is roughly the published one. This keeps the comparison honest.

Baselines

The naive reference is deterministic MPC with the full 24-hour point forecast, as in the microgrid papers. It is the controller practitioners actually deploy, and the published result is that SDDP is nearly 45% cheaper in summer and wins in 93% of summer scenarios.

The strongest existing method is pure SDDP with no online forecast information. To beat it, the hybrid must lower mean daily cost and its 95% conditional value-at-risk (the average cost of the worst 5% of days). This must hold on the same 1,000-scenario-style test set built from held-out SolarCube days, and in particular on hard-cloud days. No closed-loop numbers exist yet for SolarCube-driven microgrids; no reported result was found on Lacuna for that pairing.

We also include stochastic MPC with a fixed horizon and SDDP terminal value. Here the horizon is grid-searched on validation cost over the same set of candidate horizons and with the same validation budget given to tuning $\gamma$. This is the ablation that isolates the key ingredient, the per-decision cutoff.

A second ablation keeps the adaptive cutoff but uses independent per-lead errors instead of bootstrapped error paths. It isolates whether compounding structure matters for control. All controllers share the forecaster checkpoints, scenario count, solver and online compute limit.

The proposal succeeds if the adaptive hybrid beats the best fixed-horizon hybrid by a margin larger than the seed-to-seed variation. It also succeeds if it beats pure SDDP on hard-cloud days without losing on easy days.

Why not just…?

Why not just grid-search the planning horizon? This is the most serious threat. If the useful horizon is roughly constant across days, a single tuned cutoff captures all the benefit and our estimator adds nothing. The fixed-horizon hybrid is a first-class baseline for exactly this reason, and we will report the result even if it ties. Our reason for expecting a difference is that SolarCube's hard-cloud cases are far less predictable than its easy ones: even a strong model reaches only $R^2 = 0.646$ on hard cases, against 0.392 for simply carrying the current value forward. A gap in predictability that large implies cutoffs that move by hours between regimes.

Why not just use a non-autoregressive forecaster and avoid compounding entirely? ProbTS shows that direct forecasters avoid accumulated error, but they smooth over local variation. WEATHER-5K shows the strongest of them still misses most extremes. Skill decay with lead time remains even without compounding, so the handoff question remains too. Running the hybrid with the non-autoregressive forecaster separates the two effects. If the adaptive gain disappears there, the mechanism is compounding specifically. If it persists, the mechanism is lead-time skill decay in general, and the claim narrows accordingly.

Why not just normalize the inputs (RevIN or ensemble deduction) so the autoregressive forecaster stops drifting? Both techniques are already inside our forecasters. They remove trend and daily-curve drift, which ProbTS and the Modified Auto Regressive Technique paper confirm. They do not restore information the atmosphere has destroyed. For example, the Modified Auto Regressive Technique error still roughly doubles between 10 minutes and one hour, rising from 14.20% to 22.42% mean absolute percentage error. Normalization therefore improves the forecaster but leaves the question of when to stop trusting it.

Why not just use SDDP alone, or a static risk margin? SDDP alone discards near-term forecast skill. Static risk hedging is costly: in Optimal Dispatch of Residential Photovoltaic Inverters, a 95% CVaR constraint raised solar curtailment from 27.83 to 44.05 kWh. Pure SDDP is a baseline, and a static CVaR margin can be added to it as a variant.

Decisive Checks

Three results decide the proposal. First, the adaptive cutoff must beat the best fixed horizon, primarily on hard-cloud days. Without this, the idea reduces to tuning. Second, bootstrapped error-path scenarios must beat independent per-lead scenarios. This would be direct evidence that the independence assumption in Distributed Truncated Predictive Control costs money in closed loop. Third, the empirical cutoff $k^*(t)$ should track where the forecaster's skill score crosses zero. Checking this tells us whether the rule works for the stated reason.

Secondary metrics are battery cycling, unmet thermal comfort, and the estimator's calibration measured as realized against predicted error at each lead time.