Comparisons of Different Machine Learning-Based Rainfall–Runoff Simulations under Changing Environments

Chenliang Li, Ying Jiao, Guangyuan Kan, Xiaodi Fu, Fuxin Chai, Haijun Yu, Ke LiangWater 2024 · DOI

TL;DR

This paper investigates how machine learning models (CatBoost, MBP, LSTM) perform in simulating rainfall-runoff under changing environmental conditions. They found that simply training on the entire historical dataset leads to significant performance degradation due to data inconsistencies. However, by identifying and accounting for a detected change point in the data, staged simulations dramatically improved accuracy, with CatBoost showing superior performance in this phased approach.

What Problem Does This Paper Address?

Accurate rainfall-runoff (RR) simulation is vital for water resource management, flood prediction, and disaster reduction. However, the assumption of stable hydrological time series is increasingly challenged by climate change and human activities, leading to shifts in the underlying processes that generate runoff. This "changing environment" scenario makes traditional RR models, which often assume data consistency, less reliable. The paper aims to assess the applicability and performance of different machine learning (ML) models for RR simulation when faced with such non-stationary, inconsistent hydrological data.

Methods

The study employed a non-real-time correction modeling scheme, meaning the models predict future runoff based on past rainfall and simulated antecedent flow, rather than relying on measured prior flow. Three ML models were chosen for their suitability in time-series prediction:

  • CatBoost: A gradient boosting algorithm known for its efficiency with categorical data, fast training, and ability to handle large datasets.
  • Multi-layer BP (MBP) neural network: A foundational neural network architecture, with multiple hidden layers chosen to enhance generalization and prediction accuracy for complex relationships.
  • Long Short-Term Memory (LSTM) neural network: A type of recurrent neural network specifically designed to capture long-range dependencies in sequential data, crucial for time-series modeling.

The core methodological innovation lies in how the data was handled. Initially, models were trained and tested on the entire 1961-2000 dataset. Subsequently, the researchers identified a significant change point in the hydrological data (around 1989) using Mann-Kendall trend and change point tests. This led to a second approach: splitting the dataset into a "baseline period" (pre-change) and a "changing period" (post-change) and training separate models for each period. This "staged" simulation approach was compared against the "entire dataset" approach.

Datasets

The study utilized daily rainfall and observed flow data from the Dongwan catchment in the Yiluo River Basin, China, spanning from 1961 to 2000. Rainfall data was collected from eight stations within the catchment. A 90m resolution digital elevation model (DEM) was used for watershed delineation. The dataset was divided into training (80%) and testing (20%) sets for various simulation scenarios. Notably, the paper does not introduce a new benchmark dataset but rather analyzes the impact of data consistency issues within an existing, long-term hydrological record.

Key Insights and Ideas

  • Data Consistency is Paramount: The most striking finding is that ML models trained on a dataset with a significant change point (disrupted consistency) perform poorly when applied to the entire period without accounting for this change. This highlights that even advanced ML models struggle with non-stationarity if not explicitly addressed.
  • Change Point Detection is Crucial: Identifying the historical shift in hydrological behavior (attributed to human activities like land use changes and ecological park construction) was key. This allowed for a more realistic approach to modeling.
  • Staged Simulation Outperforms Unified: Splitting the data at the identified change point and training separate models for the baseline and changing periods dramatically improved simulation accuracy compared to using the entire dataset. This suggests that adapting models to specific environmental regimes is more effective.
  • CatBoost Shines in Phased Simulations: When using staged simulations, the CatBoost model demonstrated superior accuracy, smoother simulation processes, and better matching of observed runoff across different flow levels.
  • MBP's Generalization Strength: Interestingly, when forced to simulate the entire dataset without staging, the MBP model showed the strongest generalization capability and overall accuracy, suggesting its robustness in handling complex, albeit inconsistent, data.

Key Results

  • Entire Dataset Simulation: When trained on the full 1961-2000 dataset, all models showed good performance during training (R² > 0.9), but accuracy significantly dropped during testing (R² around 0.6), with peak flow errors exceeding 20%.
  • Staged Simulation (Baseline Period): For the 1961-1989 baseline period, all models performed well, with R² values above 0.80 and errors within 20%. CatBoost slightly outperformed MBP and LSTM.
  • Staged Simulation (Changing Period): For the 1990-2000 changing period, R² values remained high (above 0.87), and errors were generally below 20%. CatBoost and MBP showed similar performance, with CatBoost's simulated daily runoff processes closely matching observed hydrographs.
  • Cross-Sampling (Entire Dataset): When using cross-sampling (even/odd years) on the entire dataset, the MBP model showed the best performance, with higher evaluation indicators and more stable FDC performance compared to CatBoost and LSTM, suggesting its advantage in dealing with complex nonlinear problems in non-stationary data.

Limitations and Future Work

  • Data Consistency Issues: The primary limitation identified is the impact of data inconsistency due to environmental changes. The study highlights that using a single model for the entire period without accounting for these shifts leads to significant biases.
  • Future Research Directions: The authors suggest several avenues for future work:
  • Exploring a wider variety of ML models for daily RR simulation.
  • Developing methods for real-time identification of mutation points in RR simulations.
  • Applying these improved methods to a broader range of watersheds to validate their generalizability.

Summary updated

Abstract

Climate change and human activities have a great impact on the environment and have challenged the assumption of the stability of the hydrological time series and the consistency of the observed data. In order to investigate the applicability of machine learning (ML)-based rainfall–runoff (RR) simulation methods under a changing environment scenario, several ML-based RR simulation models implemented in novel continuous and non-real-time correction manners were constructed. The proposed models incorporated categorical boosting (CatBoost), a multi-hidden-layer BP neural network (MBP), and a long short-term memory neural network (LSTM) as the input–output simulators. This study focused on the Dongwan catchment of the Yiluo River Basin to carry out daily RR simulations for the purpose of verifying the model’s applicability. Model performances were evaluated based on statistical indicators such as the deterministic coefficient, peak flow error, and runoff depth error. The research findings indicated that (1) ML-based RR simulation by using a consistency-disrupted dataset exhibited significant bias. During the validation phase for the three models, the R2 index decreased to around 0.6, and the peak flow error increased to over 20%. (2) Identifying data consistency transition points through data analysis and conducting staged RR simulations before and after the transition point can improve simulation accuracy. The R2 values for all three models during both the baseline and change periods were above 0.85, with peak flow and runoff depth errors of less than 20%. Among them, the CatBoost model demonstrated superior phased simulation accuracy and smoother simulation processes and closely matched the measured runoff processes across high, medium, and low water levels, with daily runoff simulation results surpassing those of the BP neural network and LSTM models. (3) When simulating the entire dataset without staged treatment, it is impossible to achieve good simulation results by adopting uniform extraction of the training samples. Under this scenario, the MBP exhibited the strongest generalization capability, highest prediction accuracy, better algorithm stability, and superior simulation accuracy compared to the CatBoost and LSTM simulators. This study offers new ideas and methods for enhancing the runoff simulation capabilities of machine learning models in changing environments.
BibTeX
@inproceedings{li2024comparisons,
  author = {Chenliang Li and Ying Jiao and Guangyuan Kan and Xiaodi Fu and Fuxin Chai and Haijun Yu and Ke Liang},
  title = {{Comparisons of Different Machine Learning-Based Rainfall–Runoff Simulations under Changing Environments}},
  booktitle = {Water},
  year = {2024},
  doi = {10.3390/w16020302},
}
Figure