Telling Weather Drivers from Their Proxies in Net-Load Forecasting

Electricity load forecasters now have access to tens of thousands of candidate weather inputs: every variable in a gridded reanalysis or weather-forecast product, at every grid cell near a city, at every lag. Most of these inputs are near-copies of each other. Temperature at one grid cell is almost the same series as temperature at the next cell, and cloud cover tracks solar radiation. Sparse regression methods can sift a few relevant inputs out of millions using only a logarithmic number of extra samples, but that guarantee requires the inputs not to be near-duplicates. When they are, the method keeps whichever copy fits the training data slightly better. That copy is often a stand-in for the true driver rather than the driver itself, and its relationship to load can break exactly when it matters, during heatwaves and cold snaps. This proposal uses a simple fact from non-linear causal discovery to break those ties. Load responds to temperature along a U-shaped curve, because cold days bring heating and hot days bring cooling. When the response is curved like this, regressing load on the true driver leaves an error term that is statistically independent of the driver. Regressing on a noisy copy leaves an error whose spread changes with the copy's value. Where linear methods see two equally good candidates, a test for this dependence can separate driver from stand-in.

The core claim is that a two-stage estimator recovers the direct weather drivers of net load from very large, highly collinear weather fields with a sample size that grows only logarithmically in the number of candidates. Net load means demand minus rooftop solar output. The first stage is a sparse non-linear screen. The second stage is a residual-independence test that picks the true driver within each group of near-duplicates. The proposal further claims that the recovered driver set changes in a predictable way as solar penetration rises. Cloud cover and irradiance stop being mere correlates of temperature and become direct drivers of net load, because they drive solar output. That shift is where renewable intermittency enters the problem. The payoff to test is forecast robustness out of distribution, during extreme weather, and the downstream operating cost that results from it.

What already exists and what is missing

The closest work is Causal Feature Selection for Weather-Driven Residential Load Forecasting. It runs PCMCI, a time-series causal discovery method based on conditional independence tests, over ERA5 reanalysis variables for nine Southern Ontario cities. It reports that the causally selected features give the lowest MAE and MAPE, and a 4.5% to 6.5% error reduction during extreme weather. It also finds that radiation and cloud effects on demand are mediated through 2-meter temperature. That paper motivates this proposal: it shows that pruning spurious weather inputs pays off out of distribution. It also leaves three things open that this proposal addresses. It used a small, pre-chosen set of city-level variables rather than the full spatial field, so the proxy problem never arose. It gave no finite-sample guarantee. It modeled consumption rather than net load, so its mediation finding says nothing about grids with significant rooftop solar.

Two papers supply the statistical template. Learning Sparse Dynamical Systems from a Single Sample Trajectory proves that Lasso followed by a least-squares refit recovers sparse linear dynamics from one trajectory with length $O(k^2 \log(n+m))$, validated on an 800-generator swing-equation model. Thresholding Procedures for High Dimensional Variable Selection and Statistical Estimation shows that thresholding plus refitting reaches near-oracle selection under conditions where plain Lasso fails. Both results are linear, and both rely on incoherence, a condition that the candidate inputs are not too correlated with one another. The research file names the violation of incoherence by spatially collinear sensors as an open problem.

Information Theoretically Optimal Sample Complexity of Learning Dynamical Directed Acyclic Graphs gives the matching $\Theta(q \log(p/q))$ minimax rate for linear dynamical causal graphs, where $p$ is the number of nodes and $q$ the maximum number of parents. It constrains this proposal by showing that logarithmic rates are achievable for dynamical causal structure. It also assumes linearity and identical noise spectra across nodes, and the proposal drops both assumptions.

The tie-breaking test comes from Least-squares independence regression for non-linear causal inference under non-Gaussian noise. That paper's estimator, LSIR, fits a regression whose residuals are as independent of the input as possible. It measures dependence with an analytic squared-loss mutual information estimate and tunes it by cross-validation. LSIR was built to orient a single cause-effect pair. This proposal reuses its estimator for a different job: choosing among candidate parents whose causal direction is already known from physics. Load does not cause weather, so the question is not which way the arrow points but which of several near-identical inputs carries it.

UnCLe: Towards Scalable Dynamic Causal Discovery in Non-linear Temporal Systems is the strongest deep alternative for non-linear temporal causal discovery. It has no identifiability guarantee and has not been tested on energy data. SolarCAST (Solar Forecasting with Causality) adjusts for presumed cloud confounders inside a neural network, but it never outputs an explicit set of drivers.

Core Mechanism

Consider a group of candidate inputs that are nearly copies of the true driver $T$: $P_j = T + u_j$, where $u_j$ is measurement or interpolation noise. Net load is modeled as $L = f(T) + g(S) + \varepsilon$, where $f$ is the curved temperature response, $g$ is the solar contribution through irradiance $S$, and $\varepsilon$ is noise independent of the drivers.

If $f$ were linear and everything Gaussian, regressing $L$ on $P_j$ would give independent residuals too, and the copies could not be told apart from $T$. When $f$ has curvature, the residual from regressing on $P_j$ contains the term $f(T) - E[f(T)\mid P_j]$. The spread of that term changes with $P_j$. The proposal defines a dependence gap for each candidate:

$$\delta_j = I\big(L - \hat h_j(P_j),\, P_j\big) - I\big(L - \hat h_T(T),\, T\big)$$

Here $I$ is the squared-loss mutual information estimated as in LSIR, and $\hat h$ is the best non-linear regression on the indicated input. The gap is zero for the true driver and positive for copies, and it grows with the curvature of $f$ and the size of the copy noise. The theoretical aim is a support-recovery bound of the form $n \gtrsim k\log p + \log(\text{group size})/\delta_{\min}^2$. The first term comes from the sparse additive screen, through the same counting argument behind $k\log(p/k)$. The second term is the usual cost of a concentration-based test that must separate a gap of size $\delta_{\min}$ among the candidates in a group. A further requirement is to handle temporal dependence through mixing arguments of the kind used by Fattahi et al. The linear limit, where $\delta_{\min}\to 0$, correctly predicts that the tie-break fails.

Research Program

The estimator has two stages. The first stage is a group-sparse additive regression of net load on spline expansions of every candidate input, with each input-and-lag pair treated as one group. Selected groups are refit without the penalty, in the spirit of Lasso followed by least squares. The second stage clusters the surviving inputs with every unselected candidate highly correlated with them. Within each cluster it fits a non-linear regression on each member and keeps the member with the smallest residual dependence. A final refit then removes clusters whose kept member adds no conditional predictive value.

The dependence estimate uses a fixed number of kernel centers, so its cost grows linearly in sample length rather than quadratically. That avoids the HSIC scaling problem the file identifies for long grid time series. The test runs only within small clusters, never over all $p$ candidates.

The analysis has three parts. The first proves the bound above for additive models with dependent samples. The second characterizes when $\delta$ is too small to use, for example in seasons where load is nearly linear in temperature. The third predicts how the recovered driver set should shift with solar penetration, with irradiance entering directly once rooftop solar output is a meaningful share of net load.

Baselines

The naive reference point is the 7-day persistence ensemble. BuildingsBench reports that this ensemble beat pretrained deep models zero-shot on residential buildings. On commercial buildings, BuildingsBench reports a 16.96% median NRMSE for previous-day persistence against 13.31% for its Transformer-L. Any method that does not clearly beat persistence on the residential and city-level series counts as a failure.

The strongest existing method is the PCMCI causal feature set ($F_3$) from Zhang et al. It is evaluated alongside that paper's all-features ($F_1$) and mutual-information-screened ($F_2$) regimes. It is the strongest competitor because it is the only published method that uses causal selection of weather inputs to make load forecasts robust under extreme weather, and its reported gain is a 4.5% to 6.5% error reduction out of distribution. The file gives only relative reductions, so absolute MAE for $F_3$ is "no reported result found on Lacuna for IESO $F_3$ absolute MAE." For that reason the proposal reruns $F_3$ itself.

PCMCI will run on the same expanded spatial candidate pool as the proposed method. It will be tried with both its linear partial-correlation test and a kernel conditional-independence test. Every selected feature set feeds the same forecasting backbones, PatchTST, GRU, and TCN. Each backbone gets the same hyperparameter search budget and the same train, validation, and test splits, with extreme-weather windows held out.

The key ablation is stage one alone: the group-sparse additive screen with refitting but without the residual-independence tie-break. This ablation isolates whether the non-linear causal ingredient does anything beyond sparse regression.

UnCLe serves as the deep causal-discovery competitor on graph recovery, trained on the same series with a matched compute budget. Beating the field means two things. On semi-synthetic data, the method must recover the true drivers more accurately than every competitor at matched sample size. On real data, it must reduce extreme-weather MAE beyond the PCMCI $F_3$ set, with confidence intervals from a block bootstrap over test periods.

Why not just…?

Why not just use Lasso or group Lasso on all lagged weather inputs, since that already has logarithmic sample complexity? This is the most serious threat. When only lagged inputs are used and there are no hidden common causes, the support of the full regression already equals the set of direct lagged parents. So in principle sparse regression is causal selection here. The gap is purely finite-sample and collinearity-driven: under near-duplicates, incoherence fails and the selected copy is close to arbitrary. The stage-one-only ablation tests this directly. If it matches the full method on proxy recovery and extreme-weather error, the tie-break is unnecessary and the honest conclusion is that sparse non-linear regression suffices. The proposal predicts gains concentrated where $\delta$ is large, at the curved ends of the temperature response, and a tie elsewhere. The ablation is reported by season to check that prediction.

Why not just run PCMCI with a non-linear conditional-independence test? On the full spatial pool, kernel tests across thousands of candidates and lags are computationally impractical, and the file notes their quadratic-to-cubic cost. Applied only to the stage-one screened set, PCMCI with kernel tests becomes a fair competitor. It may also detect proxies, because a copy is conditionally independent of load given the true driver. The difference is that conditional-independence tests need the true driver to be in the conditioning set and need enough samples to confirm independence. The residual-dependence comparison instead ranks candidates directly. Running both on the same screened set will settle which is more sample-efficient.

Why not just hand-pick the obvious drivers, such as temperature at the city center? That choice is exactly a proxy decision made without data. The proposal includes a hand-picked baseline using the nearest-grid-cell temperature and precipitation, the drivers Zhang et al. retained. It also tests whether that choice survives once rooftop solar is added. If it does, the method's value reduces to confirming domain intuition.

Why not just feed everything to a large pretrained forecaster? The all-features regime $F_1$ was already worse in Zhang et al. BuildingsBench's foundation models lost to persistence on residential load. The proposal still includes $F_1$ with PatchTST for completeness.

Proposed Experiments

The first experiment uses semi-synthetic data with known ground truth. Real ERA5 fields over Southern Ontario keep the true collinearity of the candidate pool. Net load is generated as a known U-shaped function of temperature at one specific grid cell, plus solar output driven by irradiance at another cell. The solar term uses NREL 316-system PV profiles (Spatiotemporal Graph Neural Network for Performance Prediction of Photovoltaic Power Systems) scaled to penetration levels from 0% to 40%. The experiment measures exact-recovery probability against sample length while varying $p$ across orders of magnitude. The decisive check is whether recovery follows a sharp transition in $n/\log p$, like the transition shown by Shaikh Veedu et al. A second check is whether the tie-break's advantage disappears when $f$ is made linear, as the theory predicts.

The second experiment uses real data. The IESO residential series with ERA5 inputs are expanded to the full spatial grid. A semi-synthetic net-load variant subtracts scaled PV output. The experiment reports MAE and MAPE overall and on held-out heatwave and cold-snap windows against the baselines above. It also checks whether irradiance enters the recovered driver set as penetration rises, which tests the predicted reversal of the mediation finding.

The third experiment, a smaller downstream check, feeds the competing forecasts into the CommonPower Kerber Landnetz 30-bus case with its built-in safety projection. It measures balancing cost and the number of constraint violations. CommonPower's reported paradox, where the more accurate forecaster led to higher operating cost, means lower MAE alone is not enough. The test here is whether better extreme-weather error from correct driver selection reduces violations during exactly those windows. The file reports no cost numbers for CommonPower, so all costs are measured fresh.