Advances in Land Surface Model-based Forecasting: A comparative study of LSTM, Gradient Boosting, and Feedforward Neural Network Models as prognostic state emulators

Marieke Wesselkamp, Matthew Chantry, Ewan Pinnington, Margarita Choulga, Souhail Boussetta, Maria Kalweit, Joschka Boedecker, Carsten F. Dormann, Florian Pappenberger, Gianpaolo BalsamoarXiv 2024 · OpenReview

TL;DR

This paper introduces and compares three machine learning models (LSTM, XGBoost, and MLP) as "emulators" for a complex land surface model (ECMWF's ECLand). These emulators significantly speed up simulations, offering a faster yet reliable alternative for researchers studying land surface processes and improving weather forecasts.

What Problem Does This Paper Address?

Accurately forecasting near-surface weather is crucial, and land surface models (LSMs) play a vital role by simulating water, carbon, and energy fluxes. However, these LSMs are computationally expensive, hindering rapid experimental research and the exploration of numerous scenarios. This paper addresses this bottleneck by developing and comparing statistical surrogate models (emulators) that can mimic the behavior of a full LSM with significantly reduced computational cost.

Methods

The authors compare three machine learning approaches:

  • Long-Short Term Memory (LSTM) encoder-decoder network: This model is designed to capture long-term dependencies in time-series data, making it suitable for sequential processes like those in land surface modeling.
  • Extreme Gradient Boosting (XGBoost): A powerful gradient boosting algorithm known for its high performance on tabular data and robustness with minimal tuning.
  • Feed-forward Neural Network (MLP): A standard neural network architecture that serves as a baseline for comparison.

These models were trained to emulate key prognostic state variables of the ECMWF's Integrated Forecasting System (IFS) land surface scheme, ECLand, within a physics-informed, multi-objective framework. They also explored hyperparameter tuning and normalization strategies.

Datasets

The study used global simulation and reanalysis time series from 2010 to 2022, aggregated at a 6-hourly temporal resolution. The primary data sources were:

  • ECMWF's development cycle CY49R2: This provided the basis for simulations.
  • ERA5 meteorological reanalysis data: Used for forcing and atmospheric/surface model fields.
  • ECMWF's ECLand simulation results: These served as the dynamic prognostic state variables, acting as both input and target features.

No new benchmark datasets were introduced; the focus was on emulating an existing operational model.

Key Insights and Ideas

  • Significant Speedup: Emulators drastically reduce computational time compared to the full numerical LSM, enabling faster research and experimentation.
  • LSTM's Long-Range Prowess: When carefully tuned, the LSTM excels at continental long-range predictions, likely due to its ability to capture temporal dependencies.
  • XGBoost's Consistent Performance: XGBoost demonstrates consistently high accuracy across various tasks and variables, making it a reliable choice.
  • MLP's Efficiency Trade-off: The MLP offers an excellent balance between implementation time and accuracy, making it a practical option for rapid deployment.
  • Model-Specific Strengths: Different models show advantages for specific variables (e.g., XGBoost for soil water volume, LSTM for snow cover).

Key Results

  • All emulators achieved high accuracy, with RMSEs generally below 1.58 and MAEs below 0.84 on average for the European subset.
  • LSTM achieved the best overall scores in Europe, reducing MAE by up to 37% and RMSE by up to 42% compared to other models. It also showed the highest anomaly correlation coefficient (ACC).
  • XGBoost consistently performed well across tasks and variables, particularly in soil temperature approximation.
  • MLP provided an excellent implementation-time-accuracy trade-off, being computationally efficient.
  • The runtime reduction achieved by the emulators was significant, offering a faster, yet reliable alternative.

Limitations and Future Work

  • Generalizability: While emulators performed well on synthetic data from ECLand, future work will involve transfer learning and validation on real-world observations.
  • Hyperparameter Sensitivity: Neural networks, especially LSTMs, can be sensitive to hyperparameter choices, requiring careful tuning.
  • Spatial Autocorrelation: Neural network scores tended to degrade in deeper soil layers, while XGBoost remained more stable. This suggests potential challenges in extrapolating to different spatial scales or conditions.
  • Interpretability: While the paper focuses on performance, further investigation into the interpretability of these models and their sensitivity to input features is suggested.
  • Model Ensembles: The paper hints at the potential of combining XGBoost and neural networks for even better results, a direction for future research.

Summary updated

Abstract

Most useful weather prediction for the public is near the surface. The processes that are most relevant for near-surface weather prediction are also those that are most interactive and exhibit positive feedback or have key role in energy partitioning. Land surface models (LSMs) consider these processes together with surface heterogeneity and forecast water, carbon and energy fluxes, and coupled with an atmospheric model provide boundary and initial conditions. This numerical parametrization of atmospheric boundaries being computationally expensive, statistical surrogate models are increasingly used to accelerated progress in experimental research. We evaluated the efficiency of three surrogate models in speeding up experimental research by simulating land surface processes, which are integral to forecasting water, carbon, and energy fluxes in coupled atmospheric models. Specifically, we compared the performance of a Long-Short Term Memory (LSTM) encoder-decoder network, extreme gradient boosting, and a feed-forward neural network within a physics-informed multi-objective framework. This framework emulates key states of the ECMWF's Integrated Forecasting System (IFS) land surface scheme, ECLand, across continental and global scales. Our findings indicate that while all models on average demonstrate high accuracy over the forecast period, the LSTM network excels in continental long-range predictions when carefully tuned, the XGB scores consistently high across tasks and the MLP provides an excellent implementation-time-accuracy trade-off. The runtime reduction achieved by the emulators in comparison to the full numerical models are significant, offering a faster, yet reliable alternative for conducting numerical experiments on land surfaces.
BibTeX
@misc{wesselkamp2024advances,
  author = {Marieke Wesselkamp and Matthew Chantry and Ewan Pinnington and Margarita Choulga and Souhail Boussetta and Maria Kalweit and Joschka Boedecker and Carsten F. Dormann and Florian Pappenberger and Gianpaolo Balsamo},
  title = {{Advances in Land Surface Model-based Forecasting: A comparative study of LSTM, Gradient Boosting, and Feedforward Neural Network Models as prognostic state emulators}},
  year = {2024},
  eprint = {2407.16463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.16463},
}
Figure