Response to reviewer comments

Duncan Watson‐ParrisOpenalex Unknown 2021 · DOI · OpenAlex

Abstract

Large computer models are ubiquitous in the Earth sciences. These models often have tens or hundreds of tuneable parameters and can take thousands of core hours to run to completion while generating terabytes of output. It is becoming common practice to develop emulators as fast approximations, or surrogates, of these models in order to explore the relationships between these inputs and outputs, understand uncertainties, and generate large ensembles datasets. While the purpose of these surrogates may differ, their development is often very similar. Here we introduce ESEm: an open-source tool providing a general workflow for emulating and validating a wide variety of models and outputs. It includes efficient routines for sampling these emulators for the purpose of uncertainty quantification and model calibration. It is built on well-established, high-performance libraries to ensure robustness, extensibility and scalability. We demonstrate the flexibility of ESEm through three case studies using ESEm to reduce parametric uncertainty in a general circulation model and explore precipitation sensitivity in a cloud-resolving model and scenario uncertainty in the CMIP6 multi-model ensemble.

The Computational Wall in Earth System Modeling

Modern Earth system models (ESMs) are essential for climate projection but face a significant "computational wall." Featuring hundreds of tunable parameters and requiring thousands of CPU hours per run, exhaustive sensitivity analysis or fine-tuning against observations is often intractable. While statistical surrogates, or emulators, offer a solution by providing fast approximations, their development has historically relied on bespoke, closed-source code. This lack of standardization hinders reproducibility and raises the barrier to entry for researchers.

ESEm (Earth System Emulator) addresses this by providing an open-source Python library that standardizes the emulation workflow, bridging the gap between climate physics and machine learning.

The ESEm Framework

ESEm is designed as an object-oriented wrapper around high-performance libraries, providing a unified interface for three primary emulation engines:

  • Gaussian Processes (GPs): Utilizing GPFlow, these provide robust uncertainty estimates and scale to larger datasets via GPU acceleration.
  • Neural Networks (CNNs): Implemented via Keras to capture spatial relationships in gridded climate data.
  • Random Forests (RFs): Leveraged through scikit-learn for non-linear or discontinuous responses.
The ESEm workflow integrates data collocation via CIS with a robust emulation and validation pipeline.

The tool also includes a calibration suite featuring Approximate Bayesian Computation (ABC) and Hamiltonian Monte Carlo (HMC) for identifying plausible parameter ranges and optimal fits.

Case Studies and Applications

The flexibility of ESEm is demonstrated across diverse datasets:

  1. Aerosol-Climate Modeling: Calibrating black carbon parameters in the ECHAM6.3-HAM2.3 model against AERONET ground-based observations.
  2. Cloud-Resolving Simulations: Modeling shallow precipitation sensitivity using high-resolution data.
  3. CMIP6 ScenarioMIP: Exploring global temperature responses across various future socio-economic pathways (SSPs) using a multi-model ensemble.
CNN-based emulation of global Absorption Aerosol Optical Depth (AAOD) captures complex spatial patterns.

Key Insights and Results

ESEm's primary contribution is the standardization of the emulation workflow. By handling data loading, spatial collocation (via the Community Intercomparison Suite), training, and validation, ESEm allows scientists to focus on physical interpretation.

Key results include:

  • Parameter Pruning: In the black carbon study, ESEm ruled out 27% of the parameter space as "implausible," narrowing uncertainty in climate projections.
  • High-Fidelity Surrogates: The Random Forest emulator achieved an $R^2$ of 0.81 for shallow precipitation, capturing physical constraints like non-negativity without explicit programming.
  • Policy Relevance: The CMIP6 emulator generated joint probability distributions for warming limits, translating complex ensembles into actionable policy insights.

Limitations and Future Directions

While ESEm effectively addresses parametric uncertainty, it does not correct for structural uncertainty—fundamental flaws in model physics. Additionally, while CNNs are powerful for spatial data, they remain data-hungry compared to GPs. Future development will focus on integrating "likelihood-free" inference techniques and expanding the library of neural network architectures to handle higher-dimensional outputs.

Summary updated

Abstract

<strong class="journal-contentHeaderColor">Abstract.</strong> Large computer models are ubiquitous in the Earth sciences. These models often have tens or hundreds of tuneable parameters and can take thousands of core hours to run to completion while generating terabytes of output. It is becoming common practice to develop emulators as fast approximations, or surrogates, of these models in order to explore the relationships between these inputs and outputs, understand uncertainties, and generate large ensembles datasets. While the purpose of these surrogates may differ, their development is often very similar. Here we introduce ESEm: an open-source tool providing a general workflow for emulating and validating a wide variety of models and outputs. It includes efficient routines for sampling these emulators for the purpose of uncertainty quantification and model calibration. It is built on well-established, high-performance libraries to ensure robustness, extensibility and scalability. We demonstrate the flexibility of ESEm through three case studies using ESEm to reduce parametric uncertainty in a general circulation model and explore precipitation sensitivity in a cloud-resolving model and scenario uncertainty in the CMIP6 multi-model ensemble.
BibTeX
@inproceedings{watsonparris2021response,
  author = {Duncan Watson‐Parris},
  title = {{Response to reviewer comments}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.5194/gmd-2021-267-ac1},
}
Figure