WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting

Stephan Rasp, Soukayna Mouatadid, Peter Dueben, Sebastian Scher, Jonathan A. Weyn, Nils ThuereyarXiv 2020 · DOI · arXiv

Abstract

Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many domains. A natural question to ask is whether data-driven methods could also be used to predict global weather patterns days in advance. First studies show promise but the lack of a common dataset and evaluation metrics make inter-comparison between studies difficult. Here we present a benchmark dataset for data-driven medium-range weather forecasting, a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5 archive that has been processed to facilitate the use in machine learning models. We propose simple and clear evaluation metrics which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models. The dataset is publicly available at https://github.com/pangeo-data/WeatherBench and the companion code is reproducible with tutorials for getting started. We hope that this dataset will accelerate research in data-driven weather forecasting.

Introduction

For decades, weather forecasting has been the domain of Numerical Weather Prediction (NWP)—massive physical models that solve fluid dynamics equations on supercomputers. While highly successful, NWP is computationally expensive and struggles with certain localized phenomena. Recently, deep learning has shown promise in "learning" atmospheric dynamics directly from historical data. However, the field has faced a major hurdle: a lack of standardization. Research groups often use different subsets of data, time periods, and evaluation metrics, making it impossible to determine which architectures actually advance the state of the art.

WeatherBench addresses this by providing a "level playing field" for the community. Much like ImageNet did for computer vision, WeatherBench provides a standardized benchmark dataset and clear metrics to objectively compare data-driven machine learning models against traditional physical simulations.

Methodology and Data

The benchmark is built on the ERA5 reanalysis archive, which provides a high-quality "best guess" of the global atmosphere from 1979 to the present.

  • Data Accessibility: To make the massive raw data (terabytes) accessible to researchers with standard GPU setups, the authors provide regridded versions at lower resolutions (5.625°, 2.8125°, and 1.40525°).
  • Variables: The dataset includes 3D fields like geopotential, temperature, and humidity across 13 vertical levels, plus 2D surface fields like 2-meter temperature and precipitation.
  • Evaluation: The primary metric is latitude-weighted RMSE. Because a regular lat-lon grid has much smaller cells at the poles than at the equator, weighting by the cosine of latitude is essential to ensure every square kilometer of the Earth contributes equally to the error score.

Forecasting Paradigms: Direct vs. Iterative

The study evaluates two primary strategies for medium-range forecasting (predicting 3 to 5 days ahead):

  1. Direct Prediction: The model is trained to jump directly from the current state ($t=0$) to the target state (e.g., $t=5$ days).
  2. Iterative Prediction: The model predicts a short time step (e.g., $t=6$ hours) and is then applied recursively to reach the final target.
Comparison of direct and iterative prediction strategies for data-driven weather forecasting.

For simple ML models, direct prediction often outperforms iterative prediction. In iterative models, small errors in the 6-hour forecast can compound quickly, causing the model to diverge into physically impossible states.

Key Results and Baselines

The paper establishes a clear hierarchy of performance for predicting Z500 (geopotential height at 500 hPa, a standard measure of atmospheric flow):

  • Operational IFS (Physical Gold Standard): ~154 $m^2s^{-2}$ (3-day RMSE).
  • CNN (Direct): ~626 $m^2s^{-2}$.
  • Linear Regression (Direct): ~693 $m^2s^{-2}$.
  • Persistence: ~936 $m^2s^{-2}$.

While simple 5-layer CNNs outperform statistical baselines, a significant gap remains between basic deep learning and high-resolution physical NWP models. Furthermore, models trained solely on RMSE tend to produce "blurry" weather maps, as the model learns that predicting an average state is a safer way to minimize error than guessing a sharp, high-contrast storm front that might be slightly out of position.

Limitations and Future Work

WeatherBench highlights several areas for future innovation:

  • Spherical Geometry: Standard CNNs assume a flat 2D plane. Future work using Spherical Convolutions or mesh-based architectures could better handle the distortions at the poles.
  • Probabilistic Forecasting: This version focuses on deterministic forecasts. Real-world forecasting relies on "ensembles" to estimate uncertainty, a key area for future data-driven benchmarks.
  • Physical Constraints: Integrating physical laws (like conservation of mass) into neural architectures is a promising direction to improve model stability and realism.

Summary updated

Abstract

Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many domains. A natural question to ask is whether data-driven methods could also be used to predict global weather patterns days in advance. First studies show promise but the lack of a common dataset and evaluation metrics make inter-comparison between studies difficult. Here we present a benchmark dataset for data-driven medium-range weather forecasting, a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5 archive that has been processed to facilitate the use in machine learning models. We propose simple and clear evaluation metrics which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models. The dataset is publicly available at https://github.com/pangeo-data/WeatherBench and the companion code is reproducible with tutorials for getting started. We hope that this dataset will accelerate research in data-driven weather forecasting.
BibTeX
@misc{rasp2020weatherbench,
  author = {Stephan Rasp and Soukayna Mouatadid and Peter Dueben and Sebastian Scher and Jonathan A. Weyn and Nils Thuerey},
  title = {{WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting}},
  year = {2020},
  eprint = {2002.00469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.00469},
}
Figure