ClimaX: A foundation model for weather and climate

Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, Aditya Grover

Abstract

Traditional weather and climate modeling relies on computationally intensive numerical simulations that struggle to scale with the increasing volume of observational data. While deep learning has emerged as a fast alternative, most existing models are task-specific and lack the flexibility to generalize across different variables or spatial scales. ClimaX introduces the first foundation model for Earth sciences, utilizing a Transformer-based architecture pre-trained on massive climate simulations. By employing novel variable tokenization and aggregation techniques, ClimaX demonstrates superior performance across diverse tasks, including global forecasting, climate projection, and regional downscaling.

Introduction

For decades, weather forecasting and climate projections have been the domain of General Circulation Models (GCMs). These physics-based simulations are remarkably accurate but require massive supercomputing resources and are difficult to adapt to new data streams. In recent years, the success of foundation models in natural language processing and computer vision has inspired a shift toward a similar paradigm in meteorology.

ClimaX represents this shift. Rather than training a model for a single specific task—such as predicting temperature 24 hours in advance—ClimaX is designed as a general-purpose backbone. It is pre-trained on diverse datasets to learn the underlying "physics" of the atmosphere, allowing it to be fine-tuned for a wide array of downstream applications with minimal additional training.

Methodology

Architecture: Adapting Transformers for Meteorology

ClimaX is built upon the Vision Transformer (ViT) architecture, modified to handle the unique requirements of meteorological data. Standard ViTs process three-channel RGB images; however, weather data often involves dozens of variables (e.g., geopotential, humidity, wind speed) across multiple atmospheric pressure levels. ClimaX addresses this through two key innovations:

  1. Variable Tokenization: Each physical variable is treated as a separate modality and tokenized independently. This modular approach allows the model to ingest any combination of input variables, even if the available data changes between pre-training and fine-tuning.
  2. Variable Aggregation: To maintain computational efficiency, ClimaX uses a cross-attention mechanism to merge tokens from different variables at the same spatial location into a single unified token. This prevents the sequence length from becoming unmanageable while still capturing the complex interactions between different physical phenomena.

Pre-training and Data Strategy

A defining feature of ClimaX is its use of simulated data for pre-training. The authors utilize the CMIP6 (Coupled Model Intercomparison Project) archive, a vast collection of climate simulations from 49 modeling groups. This provides a nearly inexhaustible source of physically consistent data.

The model is trained using a self-supervised randomized forecasting objective. Instead of predicting a fixed time step, ClimaX is presented with an atmospheric snapshot and tasked with predicting the state at a random lead time (ranging from 6 to 168 hours). This objective forces the model to learn the fundamental temporal dynamics of the atmosphere rather than simply memorizing specific patterns.

Evaluation and Results

ClimaX was evaluated against industry standards and specialized models across several domains:

  • Global Weather Forecasting: When fine-tuned on the ERA5 reanalysis dataset, ClimaX matched or outperformed the Integrated Forecasting System (IFS)—the gold-standard numerical model—at lead times of seven days and beyond.
  • Climate Projection: On the ClimateBench benchmark, which evaluates long-term responses to forcing factors like $CO_2$ levels, ClimaX achieved state-of-the-art results, surpassing neural networks specifically designed for climate modeling.
  • Regional and Spatial Generality: Despite being trained on global data, ClimaX outperformed specialized regional models for North American weather forecasting (ERA5-NA). It also proved effective at "downscaling," the process of generating high-resolution details from low-resolution climate model outputs.

Key Insights

Simulation as a Proxy for Reality

The research demonstrates that pre-training on simulated climate data (CMIP6) significantly enhances performance on real-world observational data (ERA5). This suggests that the model successfully learns the "language of the atmosphere" from simulations, which then generalizes to the complexities of the real world.

Scaling Laws in Earth Science

Similar to Large Language Models (LLMs), ClimaX exhibits clear scaling laws. Performance improves consistently as model size and the volume of pre-training data increase. This indicates that the path toward more accurate weather and climate models may lie in increasing computational scale and data diversity.

Limitations and Future Directions

While ClimaX represents a significant milestone, several challenges remain. Due to hardware constraints, the current model operates at a spatial resolution of 1.4°, which is coarser than the 0.25° resolution used by the highest-fidelity operational models. Additionally, while ClimaX excels at direct forecasting, its performance in "iterative" forecasting—where the model's output is fed back as input for long-term roll-outs—requires further stabilization to prevent error accumulation.

Future iterations of ClimaX aim to incorporate unstructured data modalities, such as direct satellite observations and sparse weather station feeds, and to scale the architecture to handle higher spatial resolutions.

Conclusion

ClimaX demonstrates that the foundation model paradigm is highly effective for weather and climate science. By moving away from "one model, one task" architectures and leveraging the power of large-scale pre-training on simulations, ClimaX provides a flexible, scalable, and high-performance framework for understanding and predicting the Earth's complex atmospheric systems.

Summary updated

BibTeX
@inproceedings{nguyen2023climax,
  author = {Tung Nguyen and Johannes Brandstetter and Ashish Kapoor and Jayesh K. Gupta and Aditya Grover},
  title = {{ClimaX: A foundation model for weather and climate}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=TowCaiz7Ui},
}
Figure