Comment on hess-2024-111

Raoul Collenteur, Ezra Haaf, Mark Bakker, Tanja Liesch, Andreas Wünsch, Jenny Soonthornrangsan, Jeremy T. White, Nick Martin, Rui Hugman, Michael N. Fienen, de Sousa, Didier Vanden Berghe, Xinyang Fan, Tim Peterson, Jānis Bikše, Antoine Di Ciacca, Xinyue Wang, Yang Zheng, Maximilian Nölscher, Julian Koch, Raphael Schneider, Nikolas Benavides Höglund, Sivarama Krishna Reddy Chidepudi, Abel Henriot, Nicolas Massei, Abderrahim Jardani, Max Gustav Rudolph, Amir Rouhani, J. Jaime Gómez‐Hernández, Seifeddine Jomaa, Anna Pölz, Tim Franken, Morteza Behbooei, Jimmy Lin, Bryan A. Tolson, Rojin MeysamiOpenalex Unknown 2024 · DOI · OpenAlex

TL;DR

The 2022 groundwater modeling challenge pitted 15 international teams against each other to predict hydraulic head time series using a variety of data-driven approaches. The results reveal that while complex deep learning models offer high flexibility, simpler "lumped-parameter" models remain remarkably competitive, though nearly all methods struggle with reliable uncertainty estimation.

What Problem Does This Paper Address?

Hydraulic head—the level of groundwater in a well—is a critical indicator of water availability and ecosystem health. While traditional groundwater models rely on complex 3D physics and detailed subsurface mapping (which is often unavailable), data-driven models offer a faster, point-scale alternative.

However, the field lacks a "fair" benchmark. Most studies compare a new model against a weak baseline or use datasets tailored to a specific method's strengths. This paper reports on a "blind" challenge where different modeling communities (lumped-parameter, machine learning, and deep learning) were given the same data and asked to predict future water levels and estimate their own uncertainty.

Methods

The 15 participating teams were categorized into three groups based on their approach:

  • Lumped-parameter models (3 teams): These use physically-inspired reservoir concepts and impulse response functions (e.g., the Pastas or Gardenia frameworks). They are characterized by a small number of parameters and high interpretability.
  • Machine Learning (ML) models (4 teams): These utilized standard supervised algorithms like Random Forests (RF) and Support Vector Regression (SVR), often incorporating "feature engineering" like lagged weather variables.
  • Deep Learning (DL) models (8 teams): These employed high-complexity neural networks, including Long Short-Term Memory (LSTM) units, Convolutional Neural Networks (CNNs), and Transformers. These models are designed to learn complex, non-linear relationships directly from raw data.

The challenge required teams to calibrate their models on ~15 years of data and then provide a "blind" forecast for a 5-year validation period where only weather data was provided.

Datasets

The challenge utilized high-quality time series from four monitoring wells in distinct hydrogeological settings:

  1. Netherlands: Unconfined peat/sand aquifer in a temperate oceanic climate.
  2. Germany: Deep, confined karst aquifer in a continental climate.
  3. Sweden: Unsorted till in a subarctic climate (the most challenging site due to snow processes).
  4. USA (Connecticut): Confined bedrock aquifer where river stage data was a key driver.

Input data included precipitation, temperature, and potential evaporation. The USA site was unique in providing nearby river stage data as an additional stressor.

Key Insights and Ideas

  • The "Simplicity Paradox": One of the most striking findings was that simple lumped-parameter models performed nearly as well as state-of-the-art deep learning models in most scenarios. This suggests that for many groundwater systems, the fundamental response to rainfall and evaporation is linear enough that massive neural networks may be overkill.
  • AI Wins on Flexibility: Deep learning models excelled at the USA site. Why? Because they could easily ingest and weigh the "river stage" variable. Some lumped models failed here simply because they weren't designed to handle non-atmospheric inputs as effectively.
  • Uncertainty is the Achilles' Heel: Almost all teams struggled to provide accurate 95% prediction intervals. Most models were "overconfident," meaning the actual observed water levels fell outside the predicted range far more than 5% of the time.
  • Validation "Cliff": There was a consistent drop in performance from the calibration period to the validation period. This highlights the ever-present risk of overfitting, where a model learns the "noise" of the past rather than the "signal" of the future.

Key Results

  • Overall Performance: The average Nash-Sutcliffe Efficiency (NSE)—a measure where 1.0 is a perfect fit—was 0.64 across all wells in the validation period. Excluding the difficult Swedish well, the average rose to 0.72.
  • No Universal Winner: No single team or method performed best across all four wells. DL models took the top spots for the Netherlands and USA, while lumped models won in Germany and ML won in Sweden.
  • Efficiency: Data-driven modeling is fast. Most teams reported developing and calibrating their models in less than 8 hours per well, a fraction of the time required for traditional physics-based modeling.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Snow and Extremes: The poor performance at the Swedish site suggests that current data-driven models still struggle with complex processes like snowmelt and frozen ground without better input data.
  • Metric Sensitivity: The ranking of models changed depending on which metric was used (e.g., absolute error vs. relative fit), suggesting that future challenges should define "success" more explicitly beforehand.
  • Future Directions: The authors propose future challenges focusing on drought periods, large data gaps, and arid climates to see if these data-driven tools can hold up under more stressed conditions.

Reference: Collenteur, R. A., et al. (2024). Data-driven modeling of hydraulic head time series: results and lessons learned from the 2022 groundwater modeling challenge. Hydrology and Earth System Sciences.

Summary updated

Abstract

<strong class="journal-contentHeaderColor">Abstract.</strong> This paper presents the results of the 2022 groundwater time series modeling challenge, where 15 teams from different institutes applied various data-driven models to simulate hydraulic head time series at four monitoring wells. Three of the wells were located in Europe and one in the USA, in different hydrogeological settings but all in temperate or continental climates. Participants were provided with approximately 15 years of measured heads at (almost) regular time intervals and daily measurements of weather data starting some 10 years prior to the first head measurements and extending around 5 years after the last head measurement. The participants were asked to simulate the measured heads (the calibration period), provide a forecast for around 5 years after the last measurement (the validation period for which weather data was provided but not head measurements), and to include an uncertainty estimate. Three different groups of models were identified among the submissions: lumped-parameter models (3 teams), machine learning models (4 teams), and deep learning models (8 teams). Lumped-parameter models apply relatively simple response functions with few parameters, while the artificial intelligence models used models of varying complexity, generally with more parameters and more input, including input engineered from the provided data (e.g., multi-day averages). The models were evaluated on their performance to simulate the heads in the calibration period and the validation period. Different metrics were used to assess performance including metrics for average relative fit, average absolute fit, fit of extreme (high or low) heads, and the coverage of the uncertainty interval. For all wells, reasonable performance was obtained by at least one team from each of the three groups. However, the performance was not consistent across submissions within each groups, which implies that application of each method to individual sites requires significant effort and experience. Especially estimates of the uncertainty interval varied widely between teams, although some teams submitted confidence intervals rather than prediction intervals. There was not one team, let alone one method, that performed best for all wells and all performance metrics. Lumped-parameter models generally performed as well as artificial intelligence models, except for the well in the USA, where the lumped-parameter models did not use (or use to the full benefit) the provided river stage data, which was crucial for obtaining a good model. In conclusion, the challenge was a successful initiative to compare different models and learn from each other. Future challenges are needed to investigate, e.g., the performance of models in more variable climatic settings, to simulate head series with significant gaps, or to estimate the effect of drought periods.
BibTeX
@inproceedings{collenteur2024comment,
  author = {Raoul Collenteur and Ezra Haaf and Mark Bakker and Tanja Liesch and Andreas Wünsch and Jenny Soonthornrangsan and Jeremy T. White and Nick Martin and Rui Hugman and Michael N. Fienen and de Sousa and Didier Vanden Berghe and Xinyang Fan and Tim Peterson and Jānis Bikše and Antoine Di Ciacca and Xinyue Wang and Yang Zheng and Maximilian Nölscher and Julian Koch and Raphael Schneider and Nikolas Benavides Höglund and Sivarama Krishna Reddy Chidepudi and Abel Henriot and Nicolas Massei and Abderrahim Jardani and Max Gustav Rudolph and Amir Rouhani and J. Jaime Gómez‐Hernández and Seifeddine Jomaa and Anna Pölz and Tim Franken and Morteza Behbooei and Jimmy Lin and Bryan A. Tolson and Rojin Meysami and Nicolas Masséi},
  title = {{Comment on hess-2024-111}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.5194/hess-2024-111-rc1},
}

Loading related content...

Figure