An AI-Based OSSE Framework to Evaluate Learned Physical Consistency: A Multi-Model Study Using FourCastNet, Aurora, Prithvi WxC, and GraphCast

Yash Diggikar, Milton Halem, Aishwarya RamanarXiv 2026 · arXiv

TL;DR

This paper introduces the first AI-based framework for Observing System Simulation Experiments (OSSEs), testing whether machine learning weather models have truly learned the physical "balance" of the atmosphere. By evaluating four state-of-the-art models, the researchers found that large-scale "foundation" models like Aurora and Prithvi WxC can successfully propagate information from one variable to another—even in the tropics—confirming a 50-year-old meteorological conjecture and proving AI's potential for designing future satellite missions.

What Problem Does This Paper Address?

In traditional meteorology, we use Observing System Simulation Experiments (OSSEs) to decide where to place new satellites or sensors. These experiments involve simulating a "nature run" of the atmosphere and seeing how new observations improve a model's forecast. However, traditional physics-based models are computationally expensive.

While new AI weather models are orders of magnitude faster, we don't fully know if they've learned the underlying physics (like hydrostatic or geostrophic balance) or if they are just very good at pattern matching. If an AI model is told the "truth" about temperature, does it automatically correct its wind and pressure predictions in a physically consistent way? This paper seeks to answer that by creating a standardized "ML-OSSE" framework to evaluate the learned physical consistency of AI models.

Methods

The authors developed a direct-insertion ML-OSSE framework. Instead of using complex data assimilation machinery, they took a simplified approach:

  1. Spin-up: Run a 60-day autoregressive forecast. For the first 30 days, the model runs freely until its errors saturate.
  2. Direct Insertion: From day 30 to 60, every 6 hours, the researchers "hard-replace" a specific variable (e.g., temperature) in the model's state with the "truth" from reanalysis data.
  3. Evaluation: They measure how much this correction improves the other variables that weren't directly touched.

They tested four distinct architectures:

  • FourCastNet V2: Uses Spherical Fourier Neural Operators (SFNO).
  • GraphCast: A graph neural network (GNN) approach.
  • Aurora & Prithvi WxC: Large-scale "foundation" models based on transformer architectures.

Datasets

The study utilized two primary high-resolution reanalysis datasets to serve as the "truth":

  • ERA5: Used for training and evaluating FourCastNet V2, Aurora, and GraphCast.
  • MERRA-2: Used for Prithvi WxC. The experiments were conducted using the Earth2MIP platform for model intercomparison.

Key Insights and Ideas

  • Foundation Models Lead the Way: The transformer-based foundation models (Prithvi WxC and Aurora) showed the strongest physical coupling. Because they represent multiple atmospheric levels in a shared latent space, correcting one variable immediately informs others across the vertical column.
  • Reviving the Charney Conjecture: In 1969, meteorologist Jule Charney conjectured that temperature history could infer the entire atmospheric state, even in the tropics where traditional balance equations often fail. While some later studies doubted this, this paper's AI results confirm Charney’s original "all-latitude" conjecture.
  • Architectural Fragility: GraphCast exhibited a unique numerical instability. When forced with "perfect" temperature data at all 37 atmospheric levels, the model's feedback loop ran away, leading to massive errors. The researchers found that excluding the top 11 stratospheric levels (above 100 hPa) resolved the failure, highlighting that AI models can be sensitive to vertical forcing in ways traditional models are not.

Key Results

The study quantified the "impact" of inserting one variable on the rest of the model's state:

  • Overall Performance: Prithvi WxC showed the strongest cross-variable impact (52.2%), followed by Aurora (46.2%), FourCastNet (42.6%), and GraphCast (20.5%).
  • Wind-to-Temperature Coupling: Wind insertion was the most effective pathway for improvement. For Prithvi WxC, correcting the wind reduced surface pressure errors by over 87%.
  • Tropical Response: Aurora and Prithvi WxC maintained a positive response in tropical zonal winds (U500) when temperature was inserted, achieving ~52% and ~31% RMSE improvement respectively, far outperforming the other models.
Figure 6: Cross-variable RMSE response to temperature insertion. Note how Aurora (blue) and Prithvi WxC (pink) maintain strong performance across all latitudes, including the deep tropics (2°N).

Limitations and Future Work

  • Idealized Observations: The "direct insertion" of perfect, noise-free data is an upper-bound test. Real-world observations are sparse and noisy.
  • Deterministic Rollouts: The study used single 60-day runs. Future work should use ensembles to better quantify run-to-run variability and seasonal effects.
  • Integration with Data Assimilation: The authors suggest the next step is to move beyond "hard-replacement" and integrate these AI models into formal data assimilation systems like the Ensemble Kalman Filter (EnKF) or 4D-Var.

Summary updated

Abstract

Observing System Simulation Experiments (OSSEs) are a central methodology for evaluating future atmospheric observing systems. We present the first AI-based OSSE framework to test whether different AI weather models can reproduce comparable physical impact from new observations that traditional physics based models generate. We evaluate four distinct architectural AI models on the Charney conjecture that incomplete historical observations can infer the present state of the atmospheric. We considner- FourCastNet V2, GraphCast, Aurora and Prithvi WxC and integrate each autoregressively over a 60-day forecast horizon. After a 30-day spin-up, we insert ERA5 temperature fields to infer winds and related atmospheric fields. Similar OSSE experiments are conducted for wind insertion. We measure the performance of each model's response to observational information on a 30-day data assimilation cycle forecast with and without data insertion. Distinct architectural AI weather models respond differently to the same OSSE observations. Foundation models (i.e. Prithvi WxC and Aurora), produce the strongest impact. For example, while GraphCast is the weakest, all three of the four models capture the temperature-to-geopotential height response. The two foundational models retain a positive, non-trivial, tropical zonal wind at 500 hPa response to temperature insertion. Various data insertion experiments can vary significantly even within a given architecture. We identify an architecture-specific numerical instability in GraphCast's full 37-level configuration that produces a runaway feedback concentrated at the uppermost levels with ERA5 temperature insertion correctable by excluding levels at or below 100 hPa.
BibTeX
@misc{diggikar2026ai,
  author = {Yash Diggikar and Milton Halem and Aishwarya Raman},
  title = {{An AI-Based OSSE Framework to Evaluate Learned Physical Consistency: A Multi-Model Study Using FourCastNet, Aurora, Prithvi WxC, and GraphCast}},
  year = {2026},
  eprint = {2611.22024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22024},
}
Figure