A Bayesian Deep Learning Approach to Near-Term Climate Prediction

Xihaier Luo, Balasubramanya T. Nadiga, Yihui Ren, Ji Hwan Park, Wei Xu, Shinjae YooJournal of Advances in Modeling Earth Systems 2022 · DOI · OpenReview

TL;DR

This paper introduces a Bayesian deep learning approach for near-term climate prediction, specifically focusing on sea surface temperature (SST) in the North Atlantic. By combining a Densenet architecture with Bayesian inference, the model not only improves prediction accuracy over deterministic methods but also provides crucial estimates of predictive uncertainty, a vital component for reliable climate forecasting.

What Problem Does This Paper Address?

Accurate near-term climate prediction (interannual to decadal timescales) is crucial for various applications, from policy-making to adaptation strategies. However, current state-of-the-art climate models suffer from significant limitations, primarily model bias and "initialization shock." These issues reduce the skill of predictions, especially for phenomena driven by natural climate variability. While machine learning (ML) offers a promising avenue for creating computationally efficient surrogates for these complex models, existing ML approaches often provide deterministic predictions without quantifying the inherent uncertainties. This lack of uncertainty quantification hinders our ability to assess the reliability of these predictions and make informed decisions. This paper aims to bridge this gap by developing a probabilistic ML framework that addresses both prediction accuracy and uncertainty estimation.

Methods

The authors propose a Bayesian deep learning (BDL) approach. They start by employing a feedforward convolutional neural network with a Densenet architecture, which they found to outperform a convolutional LSTM in terms of predictive skill for SST. The core innovation lies in making this network probabilistic. They achieve this by:

  • Bayesian Formulation: Treating the network's parameters (weights and biases) as random variables with probability distributions, rather than fixed values. This allows for capturing epistemic uncertainty (uncertainty due to limited data or model limitations).
  • Probabilistic Modeling: Incorporating an additive noise term to model irreducible aleatoric uncertainty (inherent randomness in the data).
  • Stein Variational Gradient Descent (SVGD): Utilizing this nonparametric variational inference method to efficiently approximate the posterior distribution of the network parameters. SVGD works by iteratively moving a set of "particles" (representing possible parameter sets) towards the true posterior distribution, balancing exploration and exploitation.
  • Architecture: Employing a densely connected convolutional neural network (CNN) architecture with an encoder-decoder structure. This design is chosen to handle high-dimensional climate data and learn multiscale interactions effectively. They specifically avoid pooling operations, arguing they can lead to loss of distinctive features in climate data.

Datasets

The study uses the pre-industrial control simulation (piControl) from the Community Earth System Model (CESM2) as part of the CMIP6 project. This dataset provides 800 years of monthly sea surface temperature (SST) data for the North Atlantic. The data is preprocessed to remove the seasonal cycle and temporal mean, focusing on interannual variability. The dataset is split into training (1280 samples) and testing (128 samples) sets.

Key Insights and Ideas

  • Bayesian Deep Learning for Climate: The paper demonstrates that a Bayesian approach to deep learning can significantly enhance climate prediction by not only improving accuracy but also providing essential uncertainty estimates.
  • Densenet Architecture's Efficacy: A feedforward Densenet architecture shows superior performance compared to a recurrent ConvLSTM for this specific climate prediction task.
  • Uncertainty as a Predictor of Error: The study finds a correlation between the ensemble spread (a measure of uncertainty) and prediction error. While not perfect, this relationship suggests that uncertainty estimates can be used to gauge the reliability of predictions, especially when ground truth is unavailable.
  • Spatially Heterogeneous Uncertainty: Prediction uncertainty is not uniform across the domain. Regions with higher interannual variability (e.g., subpolar North Atlantic) tend to exhibit higher uncertainty, highlighting the importance of localized uncertainty assessment.
  • Computational Efficiency: The proposed DL model is significantly faster than ConvLSTM, and the BDL model shows good scalability, making it practical for large-scale climate modeling.

Key Results

  • The Bayesian deep learning (BDL) model, using a Densenet architecture, achieves better predictive skill than a deterministic ConvLSTM model for predicting North Atlantic SST.
  • The BDL model's predictions are sharper and exhibit lower errors compared to ConvLSTM, even at longer lead times (up to 12 months).
  • The BDL model provides estimates of uncertainty, which increase with prediction lead time, as expected.
  • The error-spread correlation, while modest (Spearman correlation of 0.33), indicates that ensemble spread can be used to estimate prediction error to a certain degree.
  • The DL model is approximately 20 times faster than the ConvLSTM model.

Limitations and Future Work

  • Optimal Uncertainty Estimation: The authors acknowledge that the error-spread relationship is not optimal, suggesting that further efforts are needed to drive these diagnostic relationships towards optimality for improved probabilistic ML methodology.
  • Rank Histogram Analysis: While the rank histogram diagnostic revealed slight under-dispersion of the ensemble, attempts to improve this by increasing particle numbers were unsuccessful. This suggests that the issue might be more complex and potentially related to geographical heterogeneity.
  • Future Directions: The authors mention ongoing work with attention-based methods and suggest that further analysis of uncertainty characteristics and their relationship with prediction error is a promising avenue. They also highlight the need for more sophisticated diagnostics to improve the reliability of probabilistic ML predictions.

Summary updated

Abstract

Since model bias and associated initialization shock are serious shortcomings that reduce prediction skills in state-of-the-art decadal climate prediction efforts, we pursue a complementary machine-learning-based approach to climate prediction. The example problem setting we consider consists of predicting natural variability of the North Atlantic sea surface temperature on the interannual timescale in the pre-industrial control simulation of the Community Earth System Model (CESM2). While previous works have considered the use of recurrent networks such as convolutional LSTMs and reservoir computing networks in this and other similar problem settings, we currently focus on the use of feedforward convolutional networks. In particular, we find that a feedforward convolutional network with a Densenet architecture is able to outperform a convolutional LSTM in terms of predictive skill. Next, we go on to consider a probabilistic formulation of the same network based on Stein variational gradient descent and find that in addition to providing useful measures of predictive uncertainty, the probabilistic (Bayesian) version improves on its deterministic counterpart in terms of predictive skill. Finally, we characterize the reliability of the ensemble of ML models obtained in the probabilistic setting by using analysis tools developed in the context of ensemble numerical weather prediction.
BibTeX
@article{luo2022bayesian,
  author = {Xihaier Luo and Balasubramanya T. Nadiga and Yihui Ren and Ji Hwan Park and Wei Xu and Shinjae Yoo},
  title = {{A Bayesian Deep Learning Approach to Near-Term Climate Prediction}},
  journal = {Journal of Advances in Modeling Earth Systems},
  year = {2022},
  doi = {10.1029/2022ms003058},
}
Figure