Efficient and stable coupling of the SuperdropNet deep-learning-based cloud microphysics (v0.1.0) with the ICON climate and weather model (v2.6.5)

Caroline Arnold, Shivani Sharma, Tobias Weigel, David S. GreenbergGeoscientific 2024 · DOI

Abstract

Machine learning (ML) algorithms can be used in Earth system models (ESMs) to emulate sub-grid-scale processes. Due to the statistical nature of ML algorithms and the high complexity of ESMs, these hybrid ML ESMs require careful validation. Simulation stability needs to be monitored in fully coupled simulations, and the plausibility of results needs to be evaluated in suitable experiments. We present the coupling of SuperdropNet, a machine learning model for emulating warm-rain processes in cloud microphysics, with ICON (Icosahedral Nonhydrostatic) model v2.6.5. SuperdropNet is trained on computationally expensive droplet-based simulations and can serve as an inexpensive proxy within weather prediction models. SuperdropNet emulates the collision–coalescence of rain and cloud droplets in a warm-rain scenario and replaces the collision–coalescence process in the two-moment cloud microphysics scheme. We address the technical challenge of integrating SuperdropNet, developed in Python and PyTorch, into ICON, written in Fortran, by implementing three different coupling strategies: embedded Python via the C foreign function interface (CFFI), pipes, and coupling of program components via Yet Another Coupler (YAC). We validate the emulator in the warm-bubble scenario and find that SuperdropNet runs stably within the experiment. By comparing experiment outcomes of the two-moment bulk scheme with SuperdropNet, we find that the results are physically consistent and discuss differences that are observed in several diagnostic variables. In addition, we provide a quantitative and qualitative computational benchmark for three different coupling strategies – embedded Python, coupler YAC, and pipes – and find that embedded Python is a useful software tool for validating hybrid ML ESMs.

Introduction

Modern weather and climate models, such as the Icosahedral Nonhydrostatic (ICON) model, are typically engineered in Fortran to achieve maximum computational efficiency on high-performance computing (HPC) systems. However, the rapid advancement of sub-grid process modeling—particularly in cloud microphysics—is increasingly driven by machine learning (ML) frameworks like PyTorch, which are native to Python. This creates a significant "language gap" that hinders the integration of state-of-the-art ML emulators into established Earth System Models (ESMs).

Beyond the technical challenge of inter-language communication, researchers must ensure that hybrid models remain numerically stable and physically consistent. ML models that perform well in "offline" training can often cause "online" simulations to crash or produce unphysical results. This study addresses these hurdles by integrating SuperdropNet, a PyTorch-based emulator for warm-rain processes, into ICON.

Methods and Coupling Strategies

The researchers implemented and benchmarked three distinct "Fortran–Python bridges" to facilitate the integration:

  • Embedded Python (CFFI): Utilizes the C Foreign Function Interface to execute Python code directly within the ICON process, allowing for shared memory and minimal data overhead.
  • Pipes (FIFO): Employs a worker-script approach where ICON and Python communicate via data pipes.
  • Yet Another Coupler (YAC): A standard tool for coupling disparate model components (e.g., atmosphere and ocean).

SuperdropNet was trained using data from McSnow, a high-fidelity 0D "box model" that simulates super-droplet interactions. In the hybrid setup, SuperdropNet replaces the traditional collision–coalescence module within ICON’s two-moment bulk microphysics scheme.

Integration hierarchy of SuperdropNet within the ICON model.

Key Results: Stability and Physical Consistency

The hybrid model was evaluated using the "warm-bubble" scenario, a standard test case where a pocket of warm air rises to form a cloud and eventually produces precipitation.

Performance Benchmarks

The study found that Embedded Python (CFFI) is the most efficient coupling strategy for model validation, performing nearly twice as fast as the YAC coupler and three times faster than pipes. While the ML-coupled version (24.1s) is slower than the highly optimized traditional bulk scheme (1.25s), it offers a massive potential speedup compared to the high-fidelity super-droplet models it emulates. Notably, 87% of the time in the CFFI setup was spent on neural network inference, suggesting that future optimizations should focus on ML execution (e.g., GPU acceleration) rather than data transfer.

Physical Plausibility

SuperdropNet demonstrated remarkable stability, completing the 120-minute simulation without numerical "explosions" or unphysical artifacts like negative rain values. The results were physically consistent with the traditional bulk scheme, though subtle differences emerged. For instance, SuperdropNet exhibited a slightly delayed onset of rain, likely reflecting the more detailed droplet physics captured during its training on super-droplet data.

Comparison of mean rain and snow evolution between the traditional bulk scheme and SuperdropNet.

Conclusion and Future Outlook

This research demonstrates that deep-learning-based cloud microphysics can be stably and efficiently coupled with complex ESMs like ICON. While CFFI proved superior for validation, the authors suggest that the YAC coupler might offer better robustness for large-scale, multi-component global simulations. Future work will focus on expanding the emulator's scope to include ice and snow processes and optimizing performance on heterogeneous GPU architectures to further bridge the gap between high-fidelity physics and computational feasibility.

Summary updated

Abstract

Machine learning (ML) algorithms can be used in Earth system models (ESMs) to emulate sub-grid-scale processes. Due to the statistical nature of ML algorithms and the high complexity of ESMs, these hybrid ML ESMs require careful validation. Simulation stability needs to be monitored in fully coupled simulations, and the plausibility of results needs to be evaluated in suitable experiments. We present the coupling of SuperdropNet, a machine learning model for emulating warm-rain processes in cloud microphysics, with ICON (Icosahedral Nonhydrostatic) model v2.6.5. SuperdropNet is trained on computationally expensive droplet-based simulations and can serve as an inexpensive proxy within weather prediction models. SuperdropNet emulates the collision–coalescence of rain and cloud droplets in a warm-rain scenario and replaces the collision–coalescence process in the two-moment cloud microphysics scheme. We address the technical challenge of integrating SuperdropNet, developed in Python and PyTorch, into ICON, written in Fortran, by implementing three different coupling strategies: embedded Python via the C foreign function interface (CFFI), pipes, and coupling of program components via Yet Another Coupler (YAC). We validate the emulator in the warm-bubble scenario and find that SuperdropNet runs stably within the experiment. By comparing experiment outcomes of the two-moment bulk scheme with SuperdropNet, we find that the results are physically consistent and discuss differences that are observed in several diagnostic variables. In addition, we provide a quantitative and qualitative computational benchmark for three different coupling strategies – embedded Python, coupler YAC, and pipes – and find that embedded Python is a useful software tool for validating hybrid ML ESMs.
BibTeX
@inproceedings{arnold2024efficient,
  author = {Caroline Arnold and Shivani Sharma and Tobias Weigel and David S. Greenberg},
  title = {{Efficient and stable coupling of the SuperdropNet deep-learning-based cloud microphysics (v0.1.0) with the ICON climate and weather model (v2.6.5)}},
  booktitle = {Geoscientific},
  year = {2024},
  doi = {10.5194/gmd-17-4017-2024},
}
Figure