The Compatibility between the Pangu Weather Forecasting Model and Meteorological Operational Data

Wencong Cheng, Yan Yan, Jiangjiang Xia, Qi Liu, Chang Qu, Zhigang Wang

Abstract

Recently, multiple data-driven models based on machine learning for weather forecasting have emerged. These models are highly competitive in terms of accuracy compared to traditional numerical weather prediction (NWP) systems. In particular, the Pangu-Weather model, which is open source for non-commercial use, has been validated for its forecasting performance by the European Centre for Medium-Range Weather Forecasts (ECMWF) and has recently been published in the journal "Nature". In this paper, we evaluate the compatibility of the Pangu-Weather model with several commonly used NWP operational analyses through case studies. The results indicate that the Pangu-Weather model is compatible with different operational analyses from various NWP systems as the model initial conditions, and it exhibits a relatively stable forecasting capability. Furthermore, we have verified that improving the quality of global or local initial conditions significantly contributes to enhancing the forecasting performance of the Pangu-Weather model.

Introduction

Machine learning-based Weather forecasting Models (MWMs), such as Pangu-Weather and GraphCast, have demonstrated the potential to match or exceed the performance of traditional Numerical Weather Prediction (NWP) systems. However, a significant hurdle for their operational adoption is their heavy reliance on the ERA5 reanalysis dataset for training. This raises critical questions: Can these models function effectively when initialized with data from other operational systems like the US NOAA-GFS or China's CMA-GRAPES? Furthermore, can regional improvements in initial conditions translate to better local forecasts within a global AI model?

Methodology

This study utilizes the open-source inference version of Pangu-Weather. The model was tested using initial conditions (ICs) derived from four distinct global NWP operational analyses: ECMWF-IFS, NOAA-GFS, CMA-GRAPES, and YHGSM.

To investigate local optimization, a hybrid dataset, "ecmfpadGFS," was constructed. This involved embedding high-quality ECMWF-IFS data for the East Asia region ($10^\circ \text{S}–60^\circ \text{N}, 60^\circ \text{E}–150^\circ \text{E}$) into a global GFS background. The performance of Pangu-Weather initialized with this hybrid data was then compared against the standard GFS-initialized run. Evaluation metrics included Root Mean Square Error (RMSE) and Anomaly Correlation Coefficient (ACC), with ERA5 serving as the ground truth.

Results and Compatibility Analysis

The Pangu-Weather model demonstrated remarkable "plug-and-play" capability. Despite being trained on ERA5, it maintained stable and accurate forecasting performance when initialized with diverse operational data.

Comparison of ERA5 ground truth (left column) with Pangu-Weather forecasts initialized by ERA5 (middle) and GFS (right) for various meteorological variables at a 72-hour lead time.

As shown in Figure 3, the spatial patterns of variables such as Mean Sea Level Pressure (MSLP), 2m Temperature (T2), and Geopotential Height (Z500) produced by Pangu-Weather (initialized with either ERA5 or GFS) are highly consistent with the ERA5 ground truth. Notably, Pangu-Weather initialized with GFS data often outperformed the GFS system's own forecasts, suggesting the AI has captured atmospheric dynamics more effectively than traditional NWP kernels.

In terms of computational efficiency, the model completed a single-step forecast in under 4 seconds on a single NVIDIA Titan RTX GPU, utilizing approximately 14GB of VRAM. This efficiency, combined with its data compatibility, makes it a viable candidate for integration into existing meteorological workflows.

Local Optimization and Training Bias

The "local patch" strategy proved highly effective. By improving the quality of initial conditions specifically over East Asia, the resulting regional forecast accuracy improved significantly. This suggests that meteorological centers can enhance local AI-driven forecasts without needing to overhaul global data acquisition.

However, a slight training bias was observed. The model performed optimally when initialized with ECMWF-derived data (ERA5 or IFS), with a measurable degradation when using GRAPES data. This indicates that while the model is compatible with various systems, it retains a preference for the "style" of the data it was trained on.

Conclusion

Pangu-Weather exhibits robust compatibility with various operational NWP analyses, proving its potential as a versatile tool for global weather forecasting. Its ability to serve as a "super-processor" for existing NWP data and the success of regional data enhancement strategies provide a clear path forward for operational implementation. Future research should expand the temporal scope of these evaluations to account for seasonal variability and explore the interpretability of the AI's superior dynamic modeling.

Summary updated

Abstract

Recently, multiple data-driven models based on machine learning for weather forecasting have emerged. These models are highly competitive in terms of accuracy compared to traditional numerical weather prediction (NWP) systems. In particular, the Pangu-Weather model, which is open source for non-commercial use, has been validated for its forecasting performance by the European Centre for Medium-Range Weather Forecasts (ECMWF) and has recently been published in the journal "Nature". In this paper, we evaluate the compatibility of the Pangu-Weather model with several commonly used NWP operational analyses through case studies. The results indicate that the Pangu-Weather model is compatible with different operational analyses from various NWP systems as the model initial conditions, and it exhibits a relatively stable forecasting capability. Furthermore, we have verified that improving the quality of global or local initial conditions significantly contributes to enhancing the forecasting performance of the Pangu-Weather model.
BibTeX
@misc{cheng2023compatibility,
  author = {Wencong Cheng and Yan Yan and Jiangjiang Xia and Qi Liu and Chang Qu and Zhigang Wang},
  title = {{The Compatibility between the Pangu Weather Forecasting Model and Meteorological Operational Data}},
  year = {2023},
  eprint = {2308.04460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.04460},
}

Loading related content...

Figure