Conformer: Embedding Continuous Attention in Vision Transformer for Weather Forecasting

Hira Saleem, Flora D. Salim, Cormac Purcell

TL;DR

This paper introduces STC-ViT, a novel Spatio-Temporal Continuous Vision Transformer for medium-range global weather forecasting. By integrating continuous-time dynamics via Neural Ordinary Differential Equations (Neural ODEs) with a Fourier Neural Operator (FNO) for global spatial context, STC-ViT effectively models the continuous nature of atmospheric processes, achieving competitive forecasting performance with significantly reduced computational costs compared to traditional methods.

What Problem Does This Paper Address?

Traditional weather forecasting relies on computationally intensive physics-based Numerical Weather Prediction (NWP) models. While deep learning, particularly transformer-based models, has shown promise as a surrogate, these models typically discretize spatio-temporal dimensions. This discretization hinders their ability to capture the inherent continuous evolution of atmospheric dynamics, leading to limitations in modeling complex weather patterns and requiring deep architectures with high parameter redundancy. The paper aims to bridge this gap by developing a transformer architecture that can effectively learn continuous spatio-temporal weather dynamics.

Methods

The core innovation of STC-ViT lies in its Spatio-Temporal Continuous approach. Instead of discrete layers, it models the transformer encoder as a time-conditioned vector field and uses a Neural Ordinary Differential Equation (Neural ODE) solver to integrate these dynamics over continuous time. This allows for explicit control over accuracy and latency.

Key components include:

  • Fourier Neural Operator (FNO): This component injects global, low-frequency spectral context into the model by operating in the Fourier domain. It helps capture large-scale modes and provides a smooth, topology-aware representation.
  • Spherical Harmonics (SH) Encoder: Used as a positional encoder, SH provides a topology-aware representation of patch centers on the sphere, anchoring the data-dependent features and improving coordinate awareness.
  • Continuous Transformer Encoder: The transformer blocks are parameterized as a continuous-time flow, where the dynamics are learned by a time-aware Transformer dynamical operator within the Neural ODE framework. This allows for modeling long-range dependencies and smooth spatiotemporal dynamics without requiring excessively deep stacks.
  • Fusion Mechanism: The model fuses data-dependent patch tokens, global spectral context from FNO, and topology-aware positional information from SH to form an initial latent state for the Neural ODE.

Datasets

  • ERA5 dataset: Used for training and evaluation. This dataset is provided by the European Centre for Medium-Range Weather Forecasting (ECMWF).
  • WeatherBench: A benchmark platform for evaluating data-driven weather forecasting models. STC-ViT was trained and evaluated on ERA5 data at a resolution of 5.625° (32 x 64 grid points) using this benchmark.
  • WeatherBench2: A more recent benchmark for data-driven global weather models. STC-ViT was also evaluated on this benchmark using ERA5 data at a higher resolution of 1.5° (121 x 240 grid points).

Key Insights and Ideas

  • Continuous Dynamics are Crucial: The paper strongly advocates for modeling weather as a continuous spatio-temporal process, arguing that discrete models inherently limit the ability to capture the true dynamics of atmospheric systems.
  • Shallow Networks Can Be Powerful: By leveraging continuous-time dynamics and global spectral context, STC-ViT demonstrates that shallow transformer architectures (e.g., depth 1) can achieve competitive performance, mitigating the need for deep, computationally expensive models.
  • Global Context Matters: The integration of FNO for global spectral context significantly improves performance, highlighting the importance of considering large-scale atmospheric interactions.
  • Topology Awareness is Beneficial: Using Spherical Harmonics for positional encoding ensures the model respects the spherical geometry of the Earth, leading to more robust and consistent representations.
  • ODE Solvers Impact Performance: The choice of ODE solver (e.g., RK4 vs. Euler) has a noticeable impact on forecasting accuracy, underscoring the importance of precise numerical integration for continuous-time models.

Key Results

  • Competitive Performance: STC-ViT achieves competitive forecasting performance on WeatherBench and WeatherBench 2, outperforming or matching state-of-the-art data-driven and NWP models, even when trained on lower-resolution data.
  • Efficiency: STC-ViT offers significantly lower data and compute costs. It achieves an inference speed of 0.125 seconds.
  • Depth-1 Performance: A depth-1 STC-ViT model trained on 5.625° resolution data outperforms deterministic ViT with depth 8 and Neural ODE-based architectures.
  • Scaling Analysis: Increasing transformer depth to 8 leads to a substantial improvement in performance (around 20%). Using denser temporal grids also improves accuracy, though it comes with a trade-off in training time.
  • Qualitative Results: Figures 5-8 show that STC-ViT produces visually plausible forecasts across various variables and lead times, with biases generally being smaller and more localized than in some other models.

Limitations and Future Work

  • Training Time at High Resolution: STC-ViT, being transformer-based, still faces limitations in training time when scaled to very high resolutions.
  • Deterministic Nature: The current approach is deterministic, which might lead to unrealistic results when uncertainties are present in the data.
  • Dataset Biases: The model's performance might be influenced by biases present in the training dataset (ERA5), potentially impacting generalizability to local phenomena or specific regional events.

Future work suggested by the authors includes:

  • Probabilistic Modeling: Extending STC-ViT to a probabilistic model to account for uncertainties.
  • Data Augmentation and Diverse Datasets: Exploring data augmentation techniques and integrating diverse datasets (regional, event-specific) to improve generalizability and mitigate biases.
  • High-Resolution Training and Evaluation: Scaling STC-ViT to accommodate multi-modal high-resolution training and evaluation.
  • Interpretability: Addressing the "black-box" nature of the model to gain insights into its learning mechanisms, which would be valuable for climate science.
  • Exploring Denser Temporal Grids: Further investigation into the trade-offs between accuracy and training time with denser temporal grids.

Summary updated

Abstract

Operational weather forecasting system relies on computationally expensive physics-based models. Recently, transformer based models have shown remarkable potential in weather forecasting achieving state-of-the-art results. However, transformers are discrete models which limit their ability to learn the continuous spatio-temporal features of the dynamical weather system. We address this issue with STC-ViT, a Spatio-Temporal Continuous Vision Transformer for weather forecasting. STC-ViT incorporates the continuous time Neural ODE layers with multi-head attention mechanism to learn the continuous weather evolution over time. The attention mechanism is encoded as a differentiable function in the transformer architecture to model the complex weather dynamics. We evaluate STC-ViT against a operational Numerical Weather Prediction (NWP) model and several deep learning based weather forecasting models. STC-ViT performs competitively with current data-driven methods in global forecasting while only being trained at lower resolution data and with less compute power.
BibTeX
@misc{saleem2024conformer,
  author = {Hira Saleem and Flora D. Salim and Cormac Purcell},
  title = {{Conformer: Embedding Continuous Attention in Vision Transformer for Weather Forecasting}},
  year = {2024},
  eprint = {2402.17966},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.17966},
}
Figure