TL;DR
EnergyBench is a first-of-its-kind live, "gate-aligned" benchmark that evaluates energy forecasting models in real-time against actual European market deadlines. It proves that for modern time-series foundation models, access to weather covariates is the single most important factor for accuracy—far outweighing the impact of model size.
What Problem Does This Paper Address?
In the world of energy trading, a forecast is only useful if it’s made before the market "gate" closes. However, most academic research relies on backtesting—running models on historical data. This approach is plagued by two major issues:
- Information Drift: Researchers often accidentally use "future" data, such as weather forecasts that weren't actually available at the time the trade would have been placed.
- Data Contamination: Modern "foundation models" (large models pre-trained on massive datasets) may have already "seen" the test data during their training phase, leading to artificially high performance.
EnergyBench addresses these by running models live. It freezes the information available at a specific cut-off time (11:30 CET) and requires models to submit forecasts before the real-world market gate at 12:00 CET.
Methods
The authors built a platform that fetches, freezes, and publishes a daily "information set" including:
- Historical actuals: Price, load, and generation data.
- Weather Covariates: Numerical Weather Prediction (NWP) data from three independent models (ICON-EU, GFS, and IFS).
- Fuel Costs: Derived marginal costs for gas plants.
The benchmark runs the models itself using submitted code. It evaluates 43 entries, including classical baselines, gradient-boosted trees (LightGBM), and 18 state-of-the-art time-series foundation models (like Amazon’s Chronos and Google’s TimesFM). Models are ranked using a Bradley-Terry rating on the Elo scale, the same system used to rank chess players, which accounts for the varying difficulty of different days.
Datasets
EnergyBench uses live data from:
- ENTSO-E Transparency Platform: For prices, load, and generation across six bidding zones (BE, NL, FR, DE-LU, DK1, DK2).
- National TSOs: Supplemental data from operators like Elia (Belgium) and RTE (France) to fill gaps.
- Open-Meteo: For real-time weather forecast runs.
Unlike static datasets, this benchmark grows every day, providing a continuous stream of "out-of-distribution" challenges for the models.
Key Insights and Ideas
- Weather is the Great Equalizer: The most striking finding is that providing weather covariates reduced foundation model errors by 37–51%. Without weather data, even the most advanced models struggle to beat simple baselines for wind and solar.
- Zero-Shot Success: Foundation models used "zero-shot" (without being specifically trained on this energy data) matched or outperformed custom-trained LightGBM models when given the same weather inputs.
- Architecture vs. Inputs: Model size (ranging from 3M to 331M parameters) had a surprisingly small effect on performance. In energy forecasting, what the model sees (covariates) matters much more than how big the model is.
- The Power of the Crowd: Median ensembles of multiple foundation models consistently ranked at the top of the leaderboard, proving more robust than any single model.
Key Results
- Error Reduction: On "matched races," covariates slashed errors by 64–73% for wind and 48–61% for solar.
- Leaderboard Standings: The top-ranked models were ensembles (
ens_hybridandens_fm_cov), followed by specific foundation models liket0-gamma,Timestral, andTimesFM 3.0. - The "Boss Level": Grid operators' (TSOs) own pre-gate forecasts are still the gold standard for wind and solar, with the best foundation models still trailing by 16–22% in error. However, for load forecasting, the gap is much smaller (4–6%).
Limitations and Future Work
The authors acknowledge several limitations:
- Seasonality: The current results only cover one season (Autumn 2026). Performance may shift significantly during summer peaks or winter freezes.
- Data Fragility: The benchmark relies on Open-Meteo as a single provider for weather data; outages there can affect the entire pipeline.
- Scope: While it covers six major European zones, expanding to more regions and including more complex settlement rules (like imbalance prices) is a planned next step.
EnergyBench sets a new standard for transparency in the field, moving away from "cherry-picked" backtests toward a rigorous, real-world proving ground for the next generation of energy AI.
Summary updated