Spatio-temporal Weather Forecasting and Attention Mechanism on Convolutional LSTMs

Selim Furkan Tekin, Oguzhan Karaahmetoglu, Fatih Ilhan, Ismail Balaban, Suleyman Serdar Kozat

Abstract

Traditional Numerical Weather Prediction (NWP) models are computationally intensive, often requiring hours to generate forecasts. This paper introduces the Weather Model (WM), a deep learning framework that leverages Convolutional Long Short-Term Memory (ConvLSTM) units enhanced with specialized attention and context-matching mechanisms. The WM achieves high-resolution forecasts in minutes, effectively managing the chaotic nature of atmospheric variables. Experimental results on ERA5 and WeatherBench datasets show that the WM outperforms standard AI baselines and provides stable, long-term predictions with logarithmic error growth.

Introduction

Weather forecasting is a critical yet challenging task due to the chaotic nature of atmospheric systems. While NWP models are the gold standard, their reliance on complex differential equations makes them slow for real-time applications. Deep learning alternatives have emerged, but they often struggle with capturing long-range spatio-temporal correlations and maintaining accuracy over time. The proposed Weather Model (WM) addresses these issues by focusing on spatial structure preservation and improved gradient flow.

Methodology

The core of the WM is an encoder-decoder architecture utilizing ConvLSTM cells. Unlike standard LSTMs, ConvLSTMs replace fully connected layers with convolutions, allowing the model to maintain the spatial integrity of meteorological features like pressure systems and storm fronts.

High-level encoder-decoder architecture of the Weather Model (WM).

2D Convolutional Attention

A key innovation is the 2D Convolutional Attention mechanism. Traditional attention mechanisms often flatten spatial data, losing critical local context. The WM uses 2D convolutions to generate attention maps that weigh input features (e.g., wind speed, temperature) based on their spatial relevance to the target forecast. This allows the model to dynamically prioritize specific atmospheric circulations.

Context Matcher

To mitigate the vanishing gradient problem and improve information transfer between the encoder and decoder, the authors introduced a Context Matcher. This module aggregates encoder hidden states across all time steps and aligns them with the decoder's sequence. This symmetry ensures that patterns identified early in the input sequence are effectively utilized during the generation of future states.

Detailed breakdown of the Attention mechanism and Context Matcher components.

Experimental Setup

The model was rigorously tested on two major datasets:

  • ERA5 Hourly Dataset: Focused on the Turkey region with 30km spatial resolution, including variables like geopotential and humidity.
  • WeatherBench: A global benchmark for medium-range forecasting, facilitating comparison with state-of-the-art AI and physical models.

Results and Discussion

The WM demonstrated significant performance gains over baselines such as U-Net and standard ConvLSTMs. On the ERA5 dataset, the WM achieved an RMSE of 1.22, compared to 1.33 for U-Net and 1.59 for standard ConvLSTM. On WeatherBench, it achieved an RMSE of 2.91, notably outperforming the physical IFS T42 model (3.09).

Beyond raw metrics, the model exhibited logarithmic error growth, a crucial property for stability in multi-day forecasts. Visual analysis of attention weights confirmed that the model focuses on physically significant regions, such as areas with high temperature gradients, enhancing the interpretability of its predictions.

Conclusion

The Weather Model (WM) represents a significant step forward in AI-driven meteorology, offering a balance of speed, accuracy, and interpretability. While it currently excels in short-to-medium range forecasts (up to 72 hours), future integration of physical constraints via Physics-Informed Neural Networks (PINNs) could further bridge the gap with high-end operational physical models for long-term climate prediction.

Summary updated

BibTeX
@article{tekin2024spatio,
  author = {Selim Furkan Tekin and Oguzhan Karaahmetoglu and Fatih Ilhan and Ismail Balaban and Suleyman Serdar Kozat},
  title = {{Spatio-temporal Weather Forecasting and Attention Mechanism on Convolutional LSTMs}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  doi = {10.1109/tgrs.2024.3409084},
}
Figure