WeatherDreamer: Learning a World Model from Multimodal Earth Observations

Kate Duffy, Elise Penn, Tristan Ballard, August Posch, Thomas J. VandalarXiv 2026 · arXiv

TL;DR

WeatherDreamer is a generative "world model" that learns to perform both data assimilation and weather forecasting end-to-end, directly from raw, multimodal observations. By bypassing the need for expensive, physics-based reanalysis systems at inference time, it produces high-fidelity, hourly global updates in just 90 seconds on a single GPU, outperforming operational ensembles in short-range surface forecasts.

What Problem Does This Paper Address?

The current "gold standard" for machine learning weather prediction (MLWP) relies on a hidden crutch: reanalysis. Models like GraphCast or Pangu-Weather are "forecast-only" engines; they require a highly processed starting state (an "analysis") produced by traditional Numerical Weather Prediction (NWP) systems. These traditional systems are computationally expensive, often updated only every six hours, and rely on hand-coded "observation operators" to translate raw satellite radiances into physical variables.

While some "observation-to-observation" ML models exist, they typically lack a recursive filtering cycle. They treat weather as a sequence of snapshots rather than a continuous dynamical system, making them struggle with the sparse, irregular, and delayed nature of real-world sensor data. WeatherDreamer aims to close this gap by learning the entire "predict-correct" cycle of data assimilation (DA) and forecasting together, directly from the source.

Methods

The authors adapt the Recurrent State-Space Model (RSSM) framework—originally developed for reinforcement learning agents like "Dreamer"—to the global atmosphere.

The temporal unrolling of the state-space world model, showing the predict-correct cycle.

Key technical components include:

  • Latent World Model: Instead of operating on physical grids, the model maintains a compact latent "belief state" ($z_t$) and a deterministic memory ($h_t$) on a HEALPix grid.
  • Bipartite Graph Encoder: To handle 18 different types of sensors (from point-based weather stations to satellite swaths), a graph neural network aggregates sparse observation tokens onto the fixed latent nodes.
  • Coordinate Decoders: Using Local Implicit Image Functions (LIIF), the model can "query" its latent state to predict what any sensor would see at any arbitrary latitude, longitude, altitude, or time.
  • Limb Correction: A novel feature is the explicit conditioning of decoders on satellite viewing angles. This allows the model to separate the atmospheric state from the geometric distortion (limb darkening) inherent in satellite imagery.
  • Probabilistic Forecasting: The model is trained with an "almost-fair" Continuous Ranked Probability Score (CRPS) objective, allowing it to generate calibrated ensembles by injecting spatially correlated noise into the latent dynamics.
Architectural overview showing how multimodal observations are embedded and decoded.

Datasets

WeatherDreamer is trained on four years (2020–2023) of data from 18 global observing systems, including:

  • Geostationary imagers: GOES-16/18/19, Meteosat, and GK2A.
  • Polar-orbiting sounders: ATMS, AMSU-A, and HIRS/4.
  • In situ sensors: ~11,000 surface stations (ISD) and radiosonde profiles (IGRA).
  • Specialized satellite data: ASCAT (ocean winds), GPS-RO (bending angles), and CloudSat (cloud-top height).

For evaluation, the authors used a "Set A" (mimicking operational latency) and a "Set B" to compare directly against the NCEP Gridpoint Statistical Interpolation (GSI) system.

Key Insights and Ideas

  • Learned Data Assimilation: WeatherDreamer doesn't need prescribed error covariances. It automatically learns "how much to trust" each instrument relative to its internal background state.
  • Geometry as a Feature: By treating the satellite viewing angle as a query parameter, the model learns to "undo" the slant-path atmospheric effects. This results in seamless, nadir-view global mosaics that are traditionally very difficult to produce without complex radiative transfer physics.
  • Emergent Physical Consistency: Even when certain sensors are withheld (e.g., rainfall or solar radiation), the model can infer these variables from other modalities. This suggests the latent state has captured a deep, physically consistent representation of the Earth system.
Reconstructed physical variables showing the model's ability to generate gap-filled global fields.

Key Results

  • Short-Range Skill: For 2m temperature, dew point, and 10m wind speed, WeatherDreamer’s ensemble outperforms NOAA’s Global Ensemble Forecast System (GEFS) out to 12 hours.
  • Superior Calibration: Traditional ensembles are often "under-dispersive" (too confident). WeatherDreamer achieves a spread-skill ratio of 0.63–0.78, significantly closer to the ideal value of 1.0 than GEFS (0.23–0.47) or ECMWF (0.26–0.41).
  • Efficiency: A global hourly analysis takes ~1.5 minutes on a single NVIDIA H100. A full 12-hour, 8-member ensemble forecast takes about 11 minutes.
  • Limb Darkening: The model successfully recovers 83–96% of observed limb darkening effects, enabling harmonized satellite imagery across different platforms.

Limitations and Future Work

Despite its success at the surface, WeatherDreamer currently lags behind mature DA systems like GSI in the free troposphere (the middle atmosphere). The authors note that the model's update "draws little" from upper-air observations like radio occultation, likely due to the sparsity of these signals on the current latent grid.

Furthermore, the dynamics currently operate on regional patches (~1600 km), which may limit the model's ability to represent the largest-scale, slowly evolving global circulation patterns that drive synoptic pressure changes. Future iterations will likely move toward a more global context and explore continuous, multi-day recursive cycling.

Summary updated

Abstract

Machine learning weather models now rival operational global forecasts but still start from analyses produced by separate, expensive data assimilation (DA) systems, which for global models are typically updated every six hours. Observation-to-observation models avoid this dependence, but most omit the recursive filtering cycle that lets DA systems carry a background state through sparse, irregular observations. We introduce WeatherDreamer, a world model that learns this cycle end-to-end from 18 global observing systems, without reanalysis. It infers a latent state from observations, carries it forward hour by hour with learned dynamics, corrects it each hour with new observations, and can predict what any of its observing systems would measure at any location. At held-out stations, its ensembles beat NOAA's Global Ensemble Forecast System (GEFS) for temperature, dew point and wind out to 12 h and are better calibrated than GEFS and the ECMWF ensemble. Precipitation, cloud-top height and solar irradiance, never inputs at inference, are inferred with skill matching or exceeding operational products at short leads. Against the first guess of NOAA's Gridpoint Statistical Interpolation (GSI) DA system, WeatherDreamer's short-range departures are within 10% of GSI's at surface stations and within 25% for moisture-sensitive microwave channels, but up to five times larger for temperature-sounding channels. Never given observation errors, its learned update assumes errors close to known instrument and retrieval errors of microwave sounders and scatterometer winds. WeatherDreamer also separates viewing geometry from atmospheric state, recovering 83-96% of observed limb darkening and producing seamless nadir-view imagery. Producing an hourly global analysis in about 1.5 minutes on one GPU, WeatherDreamer shows that assimilation and forecasting can be learned together directly from observations.
BibTeX
@misc{duffy2026weatherdreamer,
  author = {Kate Duffy and Elise Penn and Tristan Ballard and August Posch and Thomas J. Vandal},
  title = {{WeatherDreamer: Learning a World Model from Multimodal Earth Observations}},
  year = {2026},
  eprint = {2611.18393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18393},
}
Figure