Calibration and Uncertainty Quantification of Convective Parameters in an Idealized GCM

Oliver R. A. Dunbar, Alfredo Garbuno-Iñigo, Tapio Schneider, Andrew M. StuartJournal of Advances in Modeling Earth Systems 2021

TL;DR

This paper introduces a novel "Calibrate-Emulate-Sample" (CES) framework that significantly speeds up the process of calibrating climate model parameters and quantifying their uncertainties. By combining ensemble Kalman inversion for calibration, Gaussian processes for emulation, and MCMC for sampling, CES drastically reduces the computational cost of these tasks, making them feasible for complex climate models.

What Problem Does This Paper Address?

Climate models are essential tools for predicting future climate, but they contain parameters that represent complex, small-scale processes (like cloud formation) that cannot be directly resolved. These parameters are traditionally "tuned" manually using limited data, which is inefficient and doesn't provide a robust understanding of parameter uncertainties. Traditional Bayesian calibration methods, while theoretically sound, are computationally prohibitive for climate models due to the sheer number of model runs required. Furthermore, internal climate variability (natural chaotic fluctuations) introduces noise that can trap these traditional methods in incorrect solutions. This paper aims to bridge this gap by developing a computationally efficient and robust method for calibrating these parameters and quantifying their associated uncertainties.

Methods

The core of the paper is the Calibrate-Emulate-Sample (CES) framework, which consists of three stages:

  1. Calibrate (Ensemble Kalman Inversion - EKI): This stage uses variants of ensemble Kalman methods to find optimal parameter values by minimizing the mismatch between model-simulated climate statistics and observed data. EKI is chosen because it's a derivative-free optimization method that is robust to noise from internal climate variability and scales well to high-dimensional problems. It efficiently identifies regions of high posterior probability.
  2. Emulate (Gaussian Processes - GP): After EKI has identified promising parameter regions, Gaussian processes are used to build a "surrogate model" (an emulator) of the relationship between parameters and climate statistics. GPs are chosen because they can learn smooth functions from noisy data and also provide an estimate of their own uncertainty. This emulator is trained on the model runs performed during the calibration stage.
  3. Sample (MCMC with GP Emulator): Finally, Markov Chain Monte Carlo (MCMC) is used to sample from the posterior distribution of parameters. Crucially, instead of running the expensive climate model for each MCMC sample, the computationally cheap GP emulator is used. This dramatically reduces the computational burden of MCMC, making uncertainty quantification feasible.

Datasets

The study uses synthetic data generated from an idealized general circulation model (GCM). This "perfect-model" setting allows the authors to precisely control the ground truth and evaluate the performance of their method. The synthetic data consists of:

  • 30-day averages of free-tropospheric relative humidity.
  • 30-day averages of daily precipitation rate.
  • 30-day averages of the probability of daily precipitation exceeding a latitude-dependent 90th percentile threshold.

These statistics are chosen because they are relevant for climate predictions and are less sensitive to initial conditions than instantaneous states.

Key Insights and Ideas

  • The Power of "Calibrate-Emulate-Sample": The core insight is that by intelligently combining these three techniques, the computational bottleneck of climate model calibration and uncertainty quantification can be overcome. EKI efficiently finds the right "neighborhood" for the GP, and the GP then provides a fast, smooth surrogate for MCMC.
  • Robustness to Noise: EKI's inherent robustness to internal climate variability is a critical advantage, allowing it to find good parameter estimates even with noisy data.
  • GP as a Smoother: The GP emulator not only speeds up sampling but also smooths the objective function for MCMC, preventing it from getting stuck in local minima caused by model noise.
  • Efficient Training Point Selection: EKI's adaptive nature ensures that training points for the GP are concentrated in regions of high posterior probability, making the emulator most accurate where it matters most for uncertainty quantification.

Key Results

  • Computational Speedup: The CES approach achieves an estimated 1000-fold speedup compared to traditional brute-force MCMC methods that would require hundreds of thousands of model runs. CES requires only around 100-600 model runs.
  • Accurate Uncertainty Quantification: The MCMC sampling using the GP emulator produces posterior distributions that are good approximations of the true Bayesian posteriors.
  • Validation: The GP emulator's predictions at the true parameter values closely match the statistics generated by the GCM, demonstrating the emulator's accuracy.
  • Parameter Uncertainty in Predictions: The method successfully translates parametric uncertainty into quantified uncertainties in climate predictions, as demonstrated in an idealized global-warming experiment.

Limitations and Future Work

  • Scalability of GP: The authors acknowledge that Gaussian processes, while powerful, do not scale easily to very high-dimensional parameter spaces. This is identified as a bottleneck for applying CES to more complex models.
  • Future Work: The authors suggest exploring other machine learning techniques, such as random feature maps and deep neural networks, to improve the scalability of the emulation stage. They also mention that the CES framework can be refined and systematically developed as a general-purpose tool for uncertainty quantification. The current work is a proof-of-concept in a small parameter space.

Summary updated

Abstract

Abstract Parameters in climate models are usually calibrated manually, exploiting only small subsets of the available data. This precludes both optimal calibration and quantification of uncertainties. Traditional Bayesian calibration methods that allow uncertainty quantification are too expensive for climate models; they are also not robust in the presence of internal climate variability. For example, Markov chain Monte Carlo (MCMC) methods typically require model runs and are sensitive to internal variability noise, rendering them infeasible for climate models. Here we demonstrate an approach to model calibration and uncertainty quantification that requires only model runs and can accommodate internal climate variability. The approach consists of three stages: (a) a calibration stage uses variants of ensemble Kalman inversion to calibrate a model by minimizing mismatches between model and data statistics; (b) an emulation stage emulates the parameter‐to‐data map with Gaussian processes (GP), using the model runs in the calibration stage for training; (c) a sampling stage approximates the Bayesian posterior distributions by sampling the GP emulator with MCMC. We demonstrate the feasibility and computational efficiency of this calibrate‐emulate‐sample (CES) approach in a perfect‐model setting. Using an idealized general circulation model, we estimate parameters in a simple convection scheme from synthetic data generated with the model. The CES approach generates probability distributions of the parameters that are good approximations of the Bayesian posteriors, at a fraction of the computational cost usually required to obtain them. Sampling from this approximate posterior allows the generation of climate predictions with quantified parametric uncertainties.
BibTeX
@article{dunbar2021calibration,
  author = {Oliver R. A. Dunbar and Alfredo Garbuno-Iñigo and Tapio Schneider and Andrew M. Stuart},
  title = {{Calibration and Uncertainty Quantification of Convective Parameters in an Idealized GCM}},
  journal = {Journal of Advances in Modeling Earth Systems},
  year = {2021},
}
Figure