Seeing the Shared Error in Ensemble Flexibility Forecasts

An aggregator is a company that bundles thousands of small devices, such as home heating systems, batteries, and electric vehicles, into one large resource that can sell "flexibility" to the grid. Flexibility here means a promise to use more or less power for a while, on request. To make that promise, the aggregator has to predict, for each building, how long it could shift its consumption before someone gets too cold. A natural move is to predict with an ensemble, a committee of several different forecasting models, and treat their spread as a measure of uncertainty. This proposal argues that this spread fails in one specific way. Committee members that read the same weather forecast and learn from similar data make the same mistakes. Disagreement between members cannot reveal those shared mistakes, because a shared error cancels out of every pairwise difference. At the level of the whole portfolio, the independent mistakes average away across buildings. The shared ones are all that remain, and they are exactly what determines whether a promised megawatt actually shows up.

The idea is to split each forecast error into a shared part and a member-specific part, and to estimate each part from the data source that can actually reveal it. The member-specific part is estimated from abundant unlabeled disagreement: every timestep, on every building, with no need for ground truth. The shared part is estimated from the rare occasions when a flexibility request was actually issued and the true answer observed. The shared part is low-dimensional, so a handful of such events should be enough. The result is a portfolio bid whose safety margin tracks the error that does not cancel. It is evaluated by whether dispatches are delivered, not by per-building forecast error, which we expect to be a misleading proxy in the same way pixel error misleads image-quality evaluation.

Core Claim

Consider building $j$ and ensemble member $k$. Let $\hat\tau_{jk}$ be the member's prediction of time-to-comfort-violation under a flexibility request, and let $\tau_j$ be the true value. Write the error as $e_{jk} = \hat\tau_{jk} - \tau_j = c_j + \epsilon_{jk}$, where $c_j$ is common to all members and $\epsilon_{jk}$ is specific to member $k$. Pairwise disagreement $\hat\tau_{jk} - \hat\tau_{jl} = \epsilon_{jk} - \epsilon_{jl}$ contains no trace of $c_j$. Any label-free weighting scheme therefore sees only the member-specific covariance. Suppose the aggregate uses $J$ buildings and $K_{\text{eff}}$ effectively independent members, and suppose the common terms are correlated across buildings through weather with average pairwise correlation $\rho_c$. Then the variance of the mean per-building error is roughly

$$ \mathrm{Var}(\bar e) \approx \sigma_c^2\left(\rho_c + \frac{1-\rho_c}{J}\right) + \frac{\sigma_\epsilon^2}{J\,K_{\text{eff}}}, $$

where $\sigma_c^2$ and $\sigma_\epsilon^2$ are the common and member-specific error variances. The form follows from averaging uncorrelated terms, whose variance shrinks as $1/n$, while correlated terms do not shrink. For a large portfolio the first term dominates. It is invisible to disagreement and depends on only a few parameters, such as $\sigma_c$ and $\rho_c$ per weather regime or time of day. Our claim is that estimating these few parameters from a small number of labeled request events, and $\sigma_\epsilon$ from unlabeled disagreement, gives better-calibrated portfolio bids at a given label budget than any of three alternatives: methods that use only disagreement, methods that use only labels, or single conservative models.

Relation to Prior Work

The ensemble-redundancy literature motivates the claim. Diversity is the strength of the AI crowd uses the ambiguity decomposition, which splits ensemble error into average member error minus member diversity. It shows that tightly clustered frontier models add almost nothing to a forecasting ensemble while a decorrelated weaker model adds the most. It works on binary questions with labels, and it measures diversity rather than the portion of error that diversity cannot see. Beyond Majority Voting supplies the method we extend. It infers member accuracies from agreement patterns on unlabeled data (its OW-L and OW-I variants) and reports a 3.36% absolute MMLU gain over majority voting, but only for discrete answer choices. We carry the "estimate reliability from disagreement" idea over to continuous envelopes. We also point out where that idea is blind: disagreement cannot reveal the common-mode error. Reports that pre-training confines models to a shared basin (Neyshabur et al.) and that overlapping training data produces identical blind spots (Garmash et al.) are the language-and-vision analogues of shared weather inputs in our setting.

The flexibility-aggregation literature supplies the application and the strongest single-model baseline. Uncertainty-aware Flexibility Envelope Prediction in Buildings predicts time-to-violation for one building with a controller-agnostic switched-linear "battery" model. It wraps the prediction in risk-averse uncertainty sets built from Conditional Value at Risk (CVaR), the average of the worst $\alpha$-fraction of outcomes, with $\alpha$ as a tunable risk tolerance. It is single-building and single-model. We aggregate across a portfolio and across models, and we ask what CVaR sets miss when errors are shared. Aggregation and Disaggregation of Energetic Flexibility supplies the aggregation machinery. It represents flexibility as zonotopes, symmetric sets that can be summed exactly by concatenating generators, and it disaggregates in time independent of population size. It assumes device constraints are known exactly, which is the assumption we drop. Capturing Battery Flexibility with FlexOffers constrains the design. Its outer approximations retain more flexibility but can issue infeasible dispatch, and it retains about 61.2% of flexibility in charge/discharge switching. So the choice between inner and outer approximations is itself a reliability decision, and we must report it rather than hide it. Agent Cooperatives uses a proper scoring rule, a score that rewards honest probability reports, to elicit truthful capacity bids from agents. It treats each agent's forecast as independent, which is the assumption we test.

The perceptual-metric literature supplies the evaluation lens. PIPAL showed that for generative restorers, peak signal-to-noise ratio (PSNR), a pixel-wise error score, becomes anti-correlated with human judgment. It rewards conservative averages over what observers actually value. Kernel Density Steering shows that averaging ensemble samples collapses them to a blurry mean and that mode-seeking avoids this. We expect the same pattern in flexibility. Per-building mean absolute error in timesteps is the "pixel metric." Averaging members' envelopes is the "blur," producing a middle estimate that can be wrong in the direction that matters. The quantity that actually matters is whether the portfolio delivers what it sold. Our contribution here is not a new metric in the image sense. It is a test of whether ensemble choices ranked by mean absolute error are ranked the same way by delivered value.

Research Program

We will build a portfolio simulator on the Energym SimpleHouseRad-v0 environment used by the envelope-prediction paper, following that paper's protocol. It trains on Basel operation, with three nominal weeks and three weeks of random 1–4 hour power requests, and tests on unseen Lausanne weather. The file lists only this one environment, so we will create a heterogeneous portfolio of a few hundred buildings by varying envelope and comfort parameters and by perturbing occupant schedules. The weather forecast fed to all models is shared, and at test time it carries realistic forecast error. That shared forecast is the intended source of common-mode error, and we can dial it up or down to test the theory.

Ensemble members span deliberate diversity axes. They include the switched-linear battery model, a grey-box thermal resistance-capacitance model, a gradient-boosted regressor, and a small neural network, each trained on different data views, following the diversity recipes in the ensembling literature. Each member outputs a per-building time-to-violation for charge and discharge requests. Per-building predictions are lifted to zonotopic envelopes and summed with the zonotope method.

The estimator has two parts. We fit the member-specific covariance, and hence an effective member count $K_{\text{eff}}$, from pairwise disagreement over all unlabeled timesteps. We fit $\sigma_c$ and $\rho_c$, conditioned on a small set of features such as forecast lead time and temperature regime, from the labeled request events. The portfolio bid shrinks the aggregate envelope by a margin set from the estimated variance to hit a target failure rate. We also build a conformal variant that uses our variance estimate to normalize its nonconformity score. This separates the value of the decomposition from the value of a calibration wrapper. Economic value is computed against NordPool 2018 spot and imbalance prices from the FlexOffer benchmark. Applying Nordic prices to Swiss buildings is a stated simulation choice, not a claim about Swiss markets.

Baselines

The naive reference is the equal-weight average of member envelopes, bid with a margin set by member spread. Its bids use disagreement as if it were total uncertainty. The theory predicts it will be overconfident as the portfolio grows, and any method that fails to beat it is not worth pursuing.

The strongest existing single-model method is the CVaR envelope of Uncertainty-aware Flexibility Envelope Prediction in Buildings. On SimpleHouseRad-v0 it reports a 0.16% infeasible-prediction rate with mean absolute error of about 28 timesteps at $\alpha \approx 0.002$, and 6.09% with about 15 timesteps at $\alpha = 1.0$. We will rerun it on every building of our portfolio with the same training data, and we will sweep $\alpha$ with the same number of tuning evaluations given to our method. Beating it means that at a matched portfolio-level failure rate we commit more energy-hours of flexibility and earn more delivered value, or that at matched committed flexibility we fail less.

The strongest label-hungry competitor is split conformal calibration applied directly to the portfolio aggregate, using exactly the same labeled request events our method gets. No reported result found on Lacuna for conformal calibration of aggregated flexibility envelopes. We must show higher committed flexibility at the target failure rate when labeled events are few. We also report where the two methods converge as labels grow, since convergence there is expected.

The label-free competitor is inverse-variance weighting of members, with weights inferred from disagreement in the spirit of the OW-L variant of Beyond Majority Voting, adapted to regression. This baseline doubles as the key ablation. It is our method with the common-mode term removed. If it matches our calibration, the central claim is false.

A second ablation leaves the member pool fixed and removes shared weather-forecast error by giving each member an independently perturbed forecast. If our method's advantage disappears, the mechanism is confirmed. If the advantage persists, the gain comes from somewhere else.

Why not just…?

Why not just calibrate the portfolio aggregate with conformal prediction? This is the real threat. With enough labeled events, portfolio-level conformal captures shared error automatically, without any decomposition. Our answer rests on label scarcity. A labeled event is a real flexibility request, and those are rare. Worse, weather correlation across buildings means a single day's events are close to one effective sample, not hundreds, so the effective calibration set is small. Our method spends labels only on the few common-mode parameters. The experiment that settles this is failure rate versus number of labeled request-days for both methods. If conformal matches us at every label budget we can realistically simulate, the decomposition is not worth its complexity and we will say so.

Why not just run one well-tuned CVaR model per building? A single risk-averse model has no member-specific error to average out, so it needs a wide margin everywhere. The reported numbers show the cost: moving from a 6.09% failure rate to 0.16% nearly doubles the error in timesteps. More importantly, per-building CVaR margins do not account for correlation across buildings. Summing them can be too loose when errors are independent and too tight when a weather miss hits everyone. The matched-failure-rate comparison in the Baselines section tests this directly.

Why not just add more diverse members, or take the minimum across members? Diversity is the strength of the AI crowd shows that diversity is where ensemble gains come from. But diversity only shrinks the member-specific term. Members that share an input, here the weather forecast, share its error no matter how different their architectures are. Taking the minimum is more conservative than averaging, yet it still inherits any error common to all members, and it throws away flexibility whenever one member is idiosyncratically pessimistic. Both variants are included, and the shared-forecast ablation shows whether their failures track the common-mode term as predicted.

Proposed Experiments

The first decisive check tests the theory's prediction about scale. Hold the nominal failure target fixed and grow the portfolio from tens to hundreds of buildings. The theory predicts that the naive and disagreement-only methods will see realized failure rates climb above target as the portfolio grows, while ours stays near target. The second check is the label-budget curve against portfolio conformal described above. The third check is the metric-alignment test in the spirit of PIPAL. Across all ensemble configurations and aggregation rules, we compute the Spearman rank correlation between per-building mean absolute error and delivered value per committed megawatt-hour, where delivered value is priced with NordPool 2018 and charges imbalance prices for undelivered energy. A weak or negative correlation would show that the field's standard envelope metric misranks aggregation methods, which matters independently of our estimator. Finally, we repeat the scale and label-budget checks under inner versus outer zonotopic lifting, which shows whether the approximation choice flagged by the FlexOffer work interacts with shared-error calibration.