Abstract
The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified SwinV2 transformer on ERA5 data, and find that it attains superior forecast skill when compared against IFS. We present some ablations on key aspects of the training pipeline, exploring different loss functions, model sizes and depths, and multi-step fine-tuning to investigate their effect. We also examine the model performance with metrics beyond the typical ACC and RMSE, and investigate how the performance scales with model size.
Introduction
The field of AI weather prediction has seen a surge in models like GraphCast and Pangu-Weather achieving "super-human" forecasting skills. However, these models often rely on highly customized architectures and massive, proprietary compute budgets, creating hurdles for the broader research community. This study addresses these issues by exploring whether world-class performance can be achieved using standard vision architectures and a moderate compute budget, while systematically analyzing the impact of specific training "recipes."
Methods and Data
The researchers utilized a SwinV2 Transformer as the model backbone. Swin's "shifted windows" approach provides memory efficiency for high-resolution data. Three minimal modifications were made:
- Periodic Shifting: Adjusted window shifting for the Earth's spherical geometry.
- Non-Hierarchical Structure: Maintained constant spatial resolution across layers to preserve fine-scale features.
- Absolute Position Embeddings: Simplified the coordinate system for the fixed 0.25° resolution.
The training pipeline explored several key techniques: multi-step fine-tuning (autoregressive prediction), channel weighting (variable-specific loss importance), and latitude weighting (scaling loss by grid cell physical size). The model was trained on the ERA5 dataset at its full 0.25° resolution (~25km), covering 73 variables. Evaluation followed the Weatherbench 2 standard.
Key Results
The minimally modified SwinV2 model demonstrated exceptional performance, outperforming the traditional physics-based IFS HRES model in deterministic skill for critical variables such as geopotential height (Z500) and 2-meter temperature (T2m).
Despite its simpler architecture, the model performed on par with Pangu-Weather and remained competitive with GraphCast for short-range forecasts. Notably, the base model was trained in just 48 hours on 64 A100 GPUs, highlighting the accessibility of high-performance weather AI.
Ablation and Scaling Insights
Ablation studies revealed critical insights into model scaling and training dynamics:
- Width Over Depth: Increasing the model's "width" (embedding dimension) provided significantly better performance gains than increasing its "depth" (number of layers).
- Trick Interactions: Training techniques are not always additive. For instance, latitude weighting improved models using multi-step fine-tuning but degraded those using single-step training, suggesting that hyperparameters must be optimized for the entire pipeline.
- The Blurring Trade-off: While multi-step fine-tuning improves RMSE at long lead times, it can act as a low-pass filter, leading to "blurry" forecasts that lack sharp, physically realistic features.
Conclusion and Future Work
This work proves that state-of-the-art weather forecasting is achievable with relatively standard architectures and moderate compute resources. However, challenges remain. The trade-off between RMSE accuracy and physical "sharpness" suggests a need for generative techniques like Diffusion models or GANs. Additionally, improving "spread-skill" consistency to address model overconfidence is a key area for future research.
Summary updated