Unsupervised anomaly detection in spatio-temporal stream network sensor data

Edgar Santos–Fernández, Jay M. Ver Hoef, Erin E. Peterson, James McGree, César Villa, Catherine Leigh, Ryan Turner, Cameron Roberts, Kerrie Mengersen

Abstract

The use of in-situ digital sensors for water quality monitoring is becoming increasingly common worldwide. While these sensors provide near real-time data for science, the data are prone to technical anomalies that can undermine the trustworthiness of the data and the accuracy of statistical inferences, particularly in spatial and temporal analyses. Here we propose a framework for detecting anomalies in sensor data recorded in stream networks, which takes advantage of spatial and temporal autocorrelation to improve detection rates. The proposed framework involves the implementation of effective data imputation to handle missing data, alignment of time-series to address temporal disparities, and the identification of water quality events. We explore the effectiveness of a suite of state-of-the-art statistical methods including posterior predictive distributions, finite mixtures, and Hidden Markov Models (HMM). We showcase the practical implementation of automated anomaly detection in near-real time by employing a Bayesian recursive approach. This demonstration is conducted through a comprehensive simulation study and a practical application to a substantive case study situated in the Herbert River, located in Queensland, Australia, which flows into the Great Barrier Reef. We found that methods such as posterior predictive distributions and HMM produce the best performance in detecting multiple types of anomalies. Utilizing data from multiple sensors deployed relatively near one another enhances the ability to distinguish between water quality events and technical anomalies, thereby significantly improving the accuracy of anomaly detection. Thus, uncertainty and biases in water quality reporting, interpretation, and modelling are reduced, and the effectiveness of subsequent management actions improved.

Introduction

In the era of the "Internet of Water," thousands of in-situ digital sensors are being deployed to monitor river systems in real-time. However, these sensors are notoriously finicky. Biofouling, battery failures, and debris can create "technical anomalies"—data spikes, shifts, or drifts that mimic environmental disasters but are actually hardware glitches. Distinguishing between a genuine environmental event, such as a "first flush" after a rainstorm, and a sensor error (e.g., a fish bumping the lens) is a significant challenge for data integrity.

Traditional manual data cleaning is slow and expensive, while existing automated methods often analyze a single sensor's timeline in isolation, ignoring the spatial context of the river network. This paper presents a sophisticated Bayesian framework that leverages spatio-temporal autocorrelation to automatically detect sensor errors in near-real time without requiring massive pre-labeled training sets.

Methods

The proposed framework moves beyond simple threshold checks by incorporating the "where" and "when" of river data. The core of the approach is a Bayesian recursive spatio-temporal model using a Vector Autoregressive (VAR) structure. Crucially, it accounts for the unique topology of stream networks using "tail-up" and "tail-down" covariance structures. These structures recognize that water flows in one direction and connectivity is defined by branching river arms rather than simple Euclidean distance.

The process follows the IMPALE Framework:

  1. IMputation: Handling missing data gaps.
  2. ALignment: Syncing disparate time-series across the network.
  3. Events identification: Distinguishing natural spikes from technical errors.

Four unsupervised detection algorithms were evaluated: Posterior Predictive Distributions (PPD), Finite Mixture Models, Hidden Markov Models (HMM), and a purely temporal ARIMA benchmark.

Case Study: The Herbert River Basin

The framework was validated using a comprehensive simulation study and a real-world case study in the Herbert River basin, Queensland, Australia. This basin flows into the Great Barrier Reef, making accurate water quality monitoring critical for environmental management.

Map of the Herbert River basin showing the spatial distribution of the sensors used in the study.

The study utilized data from 10 high-end TriOS sensors and 5 low-cost BlueSpot sensors measuring Total Suspended Solids (TSS) and water levels. By utilizing data from multiple sensors deployed relatively near one another, the model "borrows strength" from neighbors to distinguish local technical anomalies from regional environmental events.

Results and Discussion

The spatio-temporal models significantly outperformed the temporal-only ARIMA benchmark. In simulations, the PPD method (with two iterations to reduce parameter bias) achieved a Matthews Correlation Coefficient (MCC) of 0.5064, compared to 0.3388 for ARIMA.

Time-series of Total Suspended Solids (TSS) across sensor locations in the Herbert River, with colored points indicating various detected anomaly types identified by the unsupervised framework.

In the Herbert River case study, the PPD method correctly identified 74% of anomalous observations with an overall accuracy of 83.6%. The framework proved particularly effective at identifying the onset of "sensor drift"—a slow deviation often caused by biofouling that is notoriously difficult to catch. Furthermore, the PPD (iter_1) method demonstrated low computational complexity, making it ideal for near-real-time deployment in resource-constrained environments.

Conclusion

By integrating spatial logic specific to stream networks with Bayesian recursive modeling, this framework reduces uncertainty and biases in water quality reporting. While challenges remain—such as handling highly skewed, non-normal data and further refining the detection of persistent anomalies—this unsupervised approach provides a robust path toward automated, trustworthy real-time water quality monitoring. Future work may involve hybrid systems that merge these statistical methods with physical, process-based hydrological models.

Summary updated

Abstract

The use of in-situ digital sensors for water quality monitoring is becoming increasingly common worldwide. While these sensors provide near real-time data for science, the data are prone to technical anomalies that can undermine the trustworthiness of the data and the accuracy of statistical inferences, particularly in spatial and temporal analyses. Here we propose a framework for detecting anomalies in sensor data recorded in stream networks, which takes advantage of spatial and temporal autocorrelation to improve detection rates. The proposed framework involves the implementation of effective data imputation to handle missing data, alignment of time-series to address temporal disparities, and the identification of water quality events. We explore the effectiveness of a suite of state-of-the-art statistical methods including posterior predictive distributions, finite mixtures, and Hidden Markov Models (HMM). We showcase the practical implementation of automated anomaly detection in near-real time by employing a Bayesian recursive approach. This demonstration is conducted through a comprehensive simulation study and a practical application to a substantive case study situated in the Herbert River, located in Queensland, Australia, which flows into the Great Barrier Reef. We found that methods such as posterior predictive distributions and HMM produce the best performance in detecting multiple types of anomalies. Utilizing data from multiple sensors deployed relatively near one another enhances the ability to distinguish between water quality events and technical anomalies, thereby significantly improving the accuracy of anomaly detection. Thus, uncertainty and biases in water quality reporting, interpretation, and modelling are reduced, and the effectiveness of subsequent management actions improved.
BibTeX
@inproceedings{santosfernandez2024unsupervised,
  author = {Edgar Santos–Fernández and Jay M. Ver Hoef and Erin E. Peterson and James McGree and César Villa and Catherine Leigh and Ryan Turner and Cameron Roberts and Kerrie Mengersen},
  title = {{Unsupervised anomaly detection in spatio-temporal stream network sensor data}},
  booktitle = {Water},
  year = {2024},
  doi = {10.1029/2023wr035707},
}
Figure