Precipitation Nowcasting Using Diffusion Transformer With Causal Attention

Chaorong Li, Xudong Ling, Yilan Xue, Wenjie Luo, Lihong Zhu, Fengqing Qin, Yaodong Zhou, Yuanyuan Huang

TL;DR

This paper introduces the Diffusion Transformer with Causal Attention (DTCA) model, a novel approach to short-term precipitation forecasting. By combining diffusion models, Transformers, and a new causal attention mechanism, DTCA significantly improves the accuracy and robustness of nowcasting, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Accurate short-term precipitation forecasting, especially for intense and rapidly developing weather events, remains a significant challenge. Traditional Numerical Weather Prediction (NWP) models struggle with the fine spatiotemporal scales and computational demands of convective precipitation. While deep learning methods have shown promise, they often fall short in capturing long-term spatiotemporal dependencies, leading to blurry predictions and a lack of interpretability. Existing extrapolation techniques also have limitations due to assumptions about atmospheric stability. This paper aims to bridge this gap by developing a model that can effectively capture complex, long-range spatiotemporal patterns in rainfall data while maintaining causal relationships between observed conditions and future forecasts.

Methods

The core of the proposed method is the Diffusion Transformer with Causal Attention (DTCA). This model leverages a Diffusion Model framework for generating realistic precipitation predictions and a Transformer architecture to handle spatiotemporal data.

Key novelties include:

  • Causal Attention Mechanism: This mechanism allows the model to establish spatiotemporal queries between conditional information (causes) and forecast results (results). It uses the rainfall sequence to be predicted as the "Query" and historical rainfall data as "Key-Value" pairs, enabling dynamic adjustments to predictions based on evolving rainfall trends.
  • Spatiotemporal Transformer Variants: The authors explore four different ways to structure the Transformer to process spatiotemporal data, finding that a "Full Joint Space-Time" approach, where all tokens participate in calculations simultaneously, yields the best results.
  • Channel-To-Batch Shift (CTBS) Operation: This technique enhances the model's ability to represent complex rainfall dynamics by redistributing data from the channel dimension to the batch dimension, creating "extended samples" that increase data diversity and feature richness.

The overall architecture follows an Encoder-Decoder structure, with an autoencoder compressing data into a latent space for diffusion model processing, and then decoding back to pixel space.

Datasets

The models were trained and evaluated on two datasets:

  • Swedish Dataset: A dataset for precipitation nowcasting.
  • MRMS Dataset: Another dataset used for evaluating precipitation forecasting.

The paper does not mention the introduction of any new benchmark datasets.

Key Insights and Ideas

  • Causal relationships are crucial: Explicitly modeling the causal link between past rainfall patterns and future predictions significantly improves accuracy and interpretability.
  • Global spatiotemporal processing is superior: Allowing all spatiotemporal tokens to interact simultaneously (Variant 1: Full Joint Space-Time) leads to better capture of complex dependencies compared to methods that separate spatial and temporal processing.
  • Channel-to-Batch Shift enhances representation: The CTBS operation, while simple, effectively increases data diversity and improves the model's ability to capture nuanced rainfall dynamics.
  • Transformer's self-attention is key: The Transformer's ability to capture long-range dependencies and retain detailed information is vital for accurate precipitation nowcasting, outperforming U-Net-based architectures in this regard.

Key Results

The DTCA model demonstrated state-of-the-art performance on both datasets:

  • Swedish Dataset: Achieved the best performance in CSI (0.06mm/h), FSS, and CRPS, with scores of 0.572, 0.911, and 0.115 respectively. It also showed competitive performance in heavy rainfall scenarios (CSI ≥ 6.3mm/h).
  • MRMS Dataset: Outperformed other models in CSI (1mm/h) with a score of 0.4235 (1.7% higher than SSLDM-ISI). For heavy rainfall (8mm/h), DTCA scored 0.0267, significantly better than competitors. It also maintained the lowest CRPS and highest FSS scores, indicating superior probabilistic and spatial accuracy.
  • Robustness: DTCA showed higher stability and robustness in long-term predictions, maintaining higher similarity with observed results even after 13 frames, unlike some other models.

Limitations and Future Work

  • Decoder Temporal Operations: The paper notes that the lack of temporal operations in the decoder might lead to some flickering phenomena in predictions.
  • Computational Efficiency: While the latent diffusion approach reduces computational cost, further improvements could be explored.
  • Future Work Suggestions: The authors suggest exploring high-quality 3D autoencoders to reduce temporal tokens and improve computational efficiency, as well as to enhance the capture of hierarchical structures and long-range dependencies in spatiotemporal data.

Summary updated

Abstract

Short-term precipitation forecasting remains challenging due to the difficulty in capturing long-term spatiotemporal dependencies. Current deep learning methods fall short in establishing effective dependencies between conditions and forecast results, while also lacking interpretability. To address this issue, we propose a precipitation nowcasting using a diffusion transformer with causal attention (DTCA) model. Our model leverages the transformer and combines causal attention mechanisms to establish spatiotemporal queries between conditional information (causes) and forecast results (results). This design enables the model to effectively capture long-term dependencies, allowing forecast results to maintain strong causal relationships with input conditions over a wide range of time and space. We explore four variants of spatiotemporal information interactions for DTCA, demonstrating that global spatiotemporal labeling interactions yield the best performance. In addition, we introduce a channel-to-batch shift (CTBS) operation to further enhance the model’s ability to represent complex rainfall dynamics. We conducted experiments on two datasets. Compared to state-of-the-art U-Net-based methods, our approach improved the critical success index (CSI) for predicting heavy precipitation by approximately 15% and 8%, respectively, achieving state-of-the-art performance. Our project is open source and available on GitHub at: https://github.com/ybu-lxd/DTCA.
BibTeX
@article{li2024precipitation,
  author = {Chaorong Li and Xudong Ling and Yilan Xue and Wenjie Luo and Lihong Zhu and Fengqing Qin and Yaodong Zhou and Yuanyuan Huang},
  title = {{Precipitation Nowcasting Using Diffusion Transformer With Causal Attention}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  doi = {10.1109/tgrs.2024.3510693},
}
Figure