Uncertainty Quantification in Machine Learning Modeling for Multi-Step Time Series Forecasting: Example of Recurrent Neural Networks in Discharge Simulations

Tianyu Song, Wei Ding, Haixing Liu, Jian Wu, Huicheng Zhou, Jinggang ChuWater 2020 · DOI

TL;DR

This paper introduces a novel framework for quantifying uncertainty in machine learning models used for multi-step time-series forecasting, specifically applied to river discharge simulations. By employing Analysis of Variance (ANOVA), the study disentangles the contributions of data partitioning, model choice, and architecture design to overall prediction uncertainty, highlighting that uncertainty quantification is crucial for reliable ML applications in hydrology.

What Problem Does This Paper Address?

Machine learning (ML) and deep learning (DL) models are increasingly used for time-series forecasting in hydrology, particularly for predicting river discharge. However, these models are often treated as "black boxes," and the uncertainty associated with their predictions is frequently overlooked. This uncertainty can stem from various sources, including how the data is split into training and testing sets (sample sets), the specific ML algorithm chosen (ML approach), and the internal structure of the model (ML architecture). Understanding and quantifying these contributions is vital for building trust and ensuring the reliability of ML-based hydrological predictions, especially for critical applications like flood forecasting. This paper aims to fill this gap by providing a systematic way to dissect and quantify these uncertainty sources.

Methods

The core of this research is a proposed framework for uncertainty quantification in multi-step time-series forecasting. This framework involves several key steps:

  • Data Partitioning: Creating multiple distinct sample sets from the historical data.
  • Model Selection: Choosing different ML algorithms, in this case, Simple Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks.
  • Architecture Design: Developing different model architectures. The study proposes two architectures: a standard stacked DL structure and a novel architecture that integrates hydrological expertise.
  • Ensemble Modeling: Combining various sample sets, ML approaches, and architectures to create a comprehensive set of models.
  • Uncertainty Quantification: Employing Analysis of Variance (ANOVA) theory to decompose the total prediction uncertainty into contributions from:
  • Sample sets (SS)
    • ML approaches (RA)
    • ML architectures (MA)
    • Interactions between these components.
  • Evaluation: Assessing model performance using standard hydrological metrics like Nash-Sutcliffe Efficiency (NSE) and Relative Peak-Discharge Error (RPE).

A key innovation is the application of ANOVA to disentangle these specific sources of uncertainty in the context of time-series forecasting, a departure from previous studies that often focused on single uncertainty sources.

Datasets

The study utilizes hourly discharge and rainfall data from the Anhe catchment in Southeastern China. The dataset spans twenty-nine years (1984-2012) and includes 75 flood events. This data was divided into five sample sets (SS1-SS5) for training and validation. The observed rainfall measurements are treated as perfect forecasts for lead times up to 10 hours. No new benchmark datasets were introduced; the focus was on applying the framework to existing hydrological data.

Key Insights and Ideas

  • Uncertainty is Multifaceted: The study strongly emphasizes that uncertainty in ML forecasting isn't a single entity but arises from a complex interplay of data, algorithm, and model structure. Ignoring any of these can lead to an incomplete understanding of prediction reliability.
  • Interactions Matter: The research highlights that the interactions between different components (e.g., how a specific sample set performs with a particular architecture) can be as significant, if not more so, than the individual contributions of each component.
  • Hydrological Expertise Enhances Models: Integrating domain knowledge (hydrology) into ML architecture design (Architecture 2) significantly improved discharge simulation performance, particularly for peak flows. This suggests that "pure" data-driven approaches can be further optimized by incorporating physical understanding.
  • LSTM's Strength: Long Short-Term Memory (LSTM) networks consistently outperformed simpler RNNs, especially for longer lead times, underscoring their capability in capturing long-term dependencies crucial for time-series forecasting.
  • Context-Dependent Uncertainty: The relative importance of each uncertainty source (SS, RA, MA) shifts depending on the lead time and the discharge quantile being predicted. This means a "one-size-fits-all" approach to uncertainty reduction is unlikely to be effective.

Key Results

  • LSTM Superiority: LSTM networks consistently showed better performance (higher NSE, lower RPE) compared to Simple RNNs across various lead times and flood events.
  • Architecture 2 Advantage: The hydrologically informed architecture (Architecture 2) demonstrated superior performance, especially in simulating peak discharges and for shorter lead times.
  • Uncertainty Source Contributions:
    • At short lead times (1-3 hours), Model Architecture (MA) was the dominant source of uncertainty, particularly when incorporating hydrological expertise.
  • As lead times increased (4-10 hours), Sample Set (SS) and RNN Approach (RA) contributions grew, while MA's contribution decreased.
  • For low discharge simulations, Sample Set partitioning was the primary driver of uncertainty, with ML approach and architecture having less impact.
  • Interactions between components contributed significantly to overall uncertainty, especially for peak discharge simulations.
  • No Single Source is Negligible: The study found that none of the investigated uncertainty sources could be dismissed as insignificant.

Limitations and Future Work

  • Limited Scope: The study focused on two specific RNN approaches (Simple RNN, LSTM) and two ML architectures. Future work could explore a wider range of ML/DL models and architectures.
  • Interaction Complexity: While the framework quantifies interactions, it doesn't deeply explain the nature of these interactions. Further research could delve into the hydrological interpretation of these interactions.
  • Hydrological Interpretation: The authors suggest future work should focus more on the hydrological interpretation of ML models, such as understanding the relationship between LSTM cell states and soil moisture content.
  • Generalizability: While the framework is presented as versatile, its application to other hydrological phenomena or different geographical regions would be valuable.

Summary updated

Abstract

As a revolutionary tool leading to substantial changes across many areas, Machine Learning (ML) techniques have obtained growing attention in the field of hydrology due to their potentials to forecast time series. Moreover, a subfield of ML, Deep Learning (DL) is more concerned with datasets, algorithms and layered structures. Despite numerous applications of novel ML/DL techniques in discharge simulation, the uncertainty involved in ML/DL modeling has not drawn much attention, although it is an important issue. In this study, a framework is proposed to quantify uncertainty contributions of the sample set, ML approach, ML architecture and their interactions to multi-step time-series forecasting based on the analysis of variance (ANOVA) theory. Then a discharge simulation, using Recurrent Neural Networks (RNNs), is taken as an example. Long Short-Term Memory (LSTM) network, a state-of-the-art DL approach, was selected due to its outstanding performance in time-series forecasting, and compared with simple RNN. Besides, novel discharge forecasting architecture is designed by combining the expertise of hydrology and stacked DL structure, and compared with conventional design. Taking hourly discharge simulations of Anhe (China) catchment as a case study, we constructed five sample sets, chose two RNN approaches and designed two ML architectures. The results indicate that none of the investigated uncertainty sources are negligible and the influence of uncertainty sources varies with lead-times and discharges. LSTM demonstrates its superiority in discharge simulations, and the ML architecture is as important as the ML approach. In addition, some of the uncertainty is attributable to interactions rather than individual modeling components. The proposed framework can both reveal uncertainty quantification in ML/DL modeling and provide references for ML approach evaluation and architecture design in discharge simulations. It indicates uncertainty quantification is an indispensable task for a successful application of ML/DL.
BibTeX
@inproceedings{song2020uncertainty,
  author = {Tianyu Song and Wei Ding and Haixing Liu and Jian Wu and Huicheng Zhou and Jinggang Chu},
  title = {{Uncertainty Quantification in Machine Learning Modeling for Multi-Step Time Series Forecasting: Example of Recurrent Neural Networks in Discharge Simulations}},
  booktitle = {Water},
  year = {2020},
  doi = {10.3390/w12030912},
}
Figure