HydroTuring: A physics probe benchmark testing mass, energy, and momentum conservation in hydrologic models

Zhi Li, Changming Li, Siavash Shams, Han Wang, Qingyi Yang, Yuanhang Liu, Taiqi Lian, Xin Lan, Shunan Zhou, Binlan Zhang, Siddik Barbhuiya, Songkun Yan, Bing Li, Jinlong Hu, Yaji Wang, Zhenjiang Wu, Mofan Zhang, Kaihao Long, Jingzhi Chen, Yuan-Heng WangarXiv 2026 · arXiv

TL;DR

HydroTuring is a new black-box benchmark suite of 37 "physics probes" designed to verify if hydrologic models—whether traditional process-based or modern AI-driven—actually obey fundamental laws of mass, energy, and momentum conservation. It reveals that high accuracy in matching historical streamflow data often masks significant physical inconsistencies, providing a much-needed "smoke test" for model reliability before deployment.

What Problem Does This Paper Address?

In hydrology, the gold standard for evaluating models has long been how well their simulated hydrographs match observed streamflow (often measured by Nash-Sutcliffe efficiency, or NSE). However, the authors demonstrate a critical flaw: a model can get the "right answer for the wrong reasons." For instance, a model that loses 15% of its water to a hidden sink can still achieve a "very good" NSE of 0.93.

This is particularly concerning for deep learning models, like LSTMs, which are increasingly used for flood forecasting but are not inherently constrained by physical laws. Because historical records only cover a tiny fraction of possible environmental regimes—the "curse of dimensionality"—models fitted only to streamflow may fail spectacularly when faced with non-stationary climates or rare extreme events.

Why hydrograph matching isn't enough to prove physical consistency.

Methods

The authors introduce HydroTuring (v0.1.0), a suite of 37 probes that treat physics as a software test suite. Key features include:

  • Black-Box Testing: Models are evaluated via a Docker-based harness. The benchmark doesn't need to see the model's code or weights; it only checks the outputs against a shared lexicon of variables.
  • Physics Probes: Each probe asks a specific physical question (e.g., "Does adding rain increase runoff?") and applies binary pass/fail criteria. The suite covers mass (24 probes), energy (7), and momentum (6).
  • Hidden Forcing: To prevent models from "memorizing" tests, forcing data is generated at runtime from recorded seeds.
  • Paired Perturbation Tests: Many probes compare a control run to a perturbed version (e.g., a 3°C temperature increase). This checks if the model's response is physically sound, even if its absolute values are calibrated.
  • Validation Gate: Every probe is validated against "exact" reference models that must pass and "deliberately broken" models that must fail.
Figure 2. Map of the 37 merged probes by law and family. Each entry is 1 probe in suite version 0.1.0. The cross-budget family needs outputs from both the water and the energy ledgers.

Datasets

Unlike traditional benchmarks that use fixed historical datasets (like CAMELS or Caravan), HydroTuring uses synthetic forcing generated at runtime. This allows for perfectly controlled experiments where the "true" physical response is known.

The authors evaluated 32 models to establish a baseline:

  • 3 Exact References: Models built to be physically perfect.
  • 7 AI-based Models: Including a standard LSTM and several "mass-conserving" perceptron architectures.
  • 22 Process-based Models: Established hydrologic models like VIC, PRMS, Noah-MP, and TOPMODEL.

Key Insights and Ideas

  • Architecture isn't Destiny: Enforcing mass conservation in a neural network's architecture (e.g., mass-conserving LSTMs) ensures that one law is obeyed, but it doesn't guarantee the model will behave physically in other ways, such as its response to warming or extreme rain.
  • The Transparency Gap: Nearly half (49%) of all model-probe pairs could not be scored because many models simply do not report the internal states (like soil moisture or heat fluxes) required to verify conservation. The authors argue that reporting these internal variables is essential for a model to be falsifiable.
  • Failures are in the Responses, not the Budgets: Most evaluated models pass long-term water-budget closure tests. The real failures appear in how they respond to perturbations and whether they respect physical state bounds (e.g., storage exceeding its physical capacity).
  • AI and Process-based Models are Similarly "Unphysical": Surprisingly, both classes of models failed their scored probes at similar rates (~24-26%), suggesting that being "process-based" doesn't automatically make a model physically consistent under all conditions.

Key Results

  • Only exact references passed everything: Out of the 28 models scored on 5 or more probes, only the 3 exact reference models passed every single one.
  • LSTM Case Study: A pre-trained flood-forecasting LSTM failed 6 of the 11 probes it could be scored on. It notably failed "resolution invariance" (giving different results for the same weather delivered at different time steps) and "extreme rain" (failing to scale runoff appropriately for massive storms).
  • Common Failure Points: The most frequent failures occurred in human_abstraction (ignoring prescribed water withdrawals) and state_bounds (internal stores going out of range).
The verdict matrix showing pass/fail results across all 32 models.

Limitations and Future Work

  • Smoke Test, Not Certification: Passing HydroTuring is a necessary but not sufficient condition for a good model. A model could be physically consistent but still inaccurate on a specific real-world basin.
  • Synthetic Catchments: Currently, probes use synthetic weather and catchments. The authors plan to introduce a "real-data track" using attributes from gauged basins (e.g., Caravan).
  • Momentum Development: Momentum is currently the least developed category, with probes focusing on steady-state consistency rather than full unsteady reach-by-reach balances.
  • Energy Data Scarcity: Very few hydrologic models currently report the energy fluxes needed for the energy probes, limiting the evaluation of this law.

Summary updated

Abstract

Hydrologic models, including deep learning systems now deployed for operational flood forecasting, are evaluated primarily by how closely their simulated hydrographs match observed streamflow. However, high hydrograph skill does not guarantee physical consistency: in a synthetic calibration experiment, a conceptual model that loses 15% of its water input to an unreported sink still reaches a Nash-Sutcliffe efficiency of 0.93, and no shared benchmark tests whether models obey fundamental conservation laws. Here we introduce HydroTuring (v0.1.0), a black-box benchmark of 37 physics probes that test mass (24 probes), energy (7) and momentum (6) conservation in hydrologic models. Each probe runs a controlled experiment on forcing generated at run time from a recorded seed and applies binary pass or fail criteria validated against 18 physically exact and 59 deliberately broken reference models. Across 32 evaluated models (3 exact references, 7 AI-based models and 22 process-based models), only the 3 exact references pass every probe among the 28 models scored on 5 or more probes. Process-based and AI-based models pass similar shares of their scored probes (76% and 74%), and a pre-trained flood-forecasting LSTM fails 6 of the 11 runoff-only probes it can be scored on. Furthermore, 49% of all 1,184 model-probe pairs remain unscored because models omit the required state or flux variables, while the scored failures occur in perturbation responses and state bounds rather than long-term water-budget closure. HydroTuring provides a reproducible smoke test for model development, peer review and operational deployment under an open community protocol.
BibTeX
@misc{li2026hydroturing,
  author = {Zhi Li and Changming Li and Siavash Shams and Han Wang and Qingyi Yang and Yuanhang Liu and Taiqi Lian and Xin Lan and Shunan Zhou and Binlan Zhang and Siddik Barbhuiya and Songkun Yan and Bing Li and Jinlong Hu and Yaji Wang and Zhenjiang Wu and Mofan Zhang and Kaihao Long and Jingzhi Chen and Yuan-Heng Wang},
  title = {{HydroTuring: A physics probe benchmark testing mass, energy, and momentum conservation in hydrologic models}},
  year = {2026},
  eprint = {2611.11563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11563},
}
Figure