A framework for automated anomaly detection in high frequency\n water-quality data from in situ sensors

Catherine Leigh, Omar Alsibai, Rob J. Hyndman, Sevvandi Kandanaarachchi, Olivia C. King, James McGree, Catherine Neelamraju, Jennifer Strauss, Priyanga Dilini Talagala, Ryan S. Turner, Kerrie Mengersen, Erin E. Peterson

Introduction

High-frequency in situ sensors have transformed river monitoring, providing near-continuous data on critical parameters like turbidity and conductivity. However, this "Big Data" revolution brings a significant challenge: the volume and velocity of data exceed human capacity for manual verification. Furthermore, technical glitches—ranging from battery failure to biofouling—often produce anomalies that can be mistaken for ecological events. To address this, we present a robust 10-step framework designed to automate anomaly detection, ensuring data integrity for environmental policy and management.

The 10-Step Framework

The proposed framework moves systematically from identifying end-user needs to the final deployment of detection algorithms. A critical first step is defining what constitutes an "anomaly" for a specific application, as the requirements of a researcher may differ significantly from those of a water utility manager. By ranking anomaly importance, analysts can select the most appropriate detection methods. This structured approach ensures that the technical solution aligns with the practical needs of the data consumers.

Detection Methodologies

The study evaluated three primary categories of Anomaly Detection (AD) methods:

  1. Automated Classification Rules: These utilize simple "if-then" logic to identify impossible values (e.g., negative turbidity) or gaps in data. These are highly effective for catching "Class 2" anomalies where data falls outside physical limits.
  2. Regression-Based Methods: Models such as Autoregressive Integrated Moving Average (ARIMA) predict future data points based on historical trends. Observations falling outside a 99% prediction interval are flagged. The study also tested Anomaly Detection and Mitigation (ADAM), which replaces flagged points with forecasts to prevent error propagation.
  3. Feature-Based Methods: Unsupervised algorithms like k-nearest neighbor (kNN) and HDoutliers analyze data in a transformed feature space. These methods identify observations that are "distant" from the norm without requiring explicit training or optimization.

Experimental Evaluation

The framework was tested using a year of data (March 2017 – March 2018) from two distinct sites in Queensland, Australia: the Pioneer River and Sandy Creek. Experts manually labeled over 2,000 observations to create a "ground truth" benchmark for turbidity and conductivity.

The "Covariate Curse"

A surprising finding was that incorporating additional variables (covariates), such as using river level to predict turbidity, actually degraded model performance. The complex, non-linear relationships between water-quality variables often introduced more noise than clarity, suggesting that univariate models are frequently more robust for real-time applications.

Mitigation and False Positives

While the ADAM approach (replacing anomalies with forecasts) was highly effective at identifying long-term issues like sensor drift and periods of high variability, it significantly increased false positive rates. Once a model begins relying on its own generated forecasts to mitigate errors, it can inadvertently drift away from the actual environmental signal, flagging normal data as anomalous.

The Reliability of Feature-Based Methods

Feature-based methods proved to be effective despite requiring no training. Although they missed subtle anomalies (resulting in higher false negatives), they were remarkably reliable at catching major spikes without generating false alarms. This low false-positive rate makes them attractive for systems where "crying wolf" must be avoided to maintain user trust.

Results and Recommendations

The study demonstrated that no single algorithm is a universal solution:

  • Rule-Based methods were 100% effective for catching impossible values with zero false positives.
  • Regression (ARIMA) was the most versatile for sudden spikes and level shifts, achieving accuracy scores above 0.80.
  • Feature-Based methods provided the lowest false positive rates across the board but struggled with subtle data shifts, sometimes missing hundreds of instances.

Ultimately, a hybrid approach is recommended. Combining simple rules for impossible values with regression-based models for sudden changes provides the most reliable results for real-time monitoring.

Conclusion

The success of automated anomaly detection depends heavily on the initial communication between end-users and analysts. While technical challenges like seasonality and site-specific tuning remain, this 10-step framework provides a scalable path forward for ensuring the reliability of high-frequency environmental data. Future work will explore the use of Hidden Markov Models to better account for the distinct "regimes" of wet and dry seasons, potentially improving detection during transition periods.

Summary updated

Abstract

River water-quality monitoring is increasingly conducted using automated in\nsitu sensors, enabling timelier identification of unexpected values. However,\nanomalies caused by technical issues confound these data, while the volume and\nvelocity of data prevent manual detection. We present a framework for automated\nanomaly detection in high-frequency water-quality data from in situ sensors,\nusing turbidity, conductivity and river level data. After identifying end-user\nneeds and defining anomalies, we ranked their importance and selected suitable\ndetection methods. High priority anomalies included sudden isolated spikes and\nlevel shifts, most of which were classified correctly by regression-based\nmethods such as autoregressive integrated moving average models. However, using\nother water-quality variables as covariates reduced performance due to complex\nrelationships among variables. Classification of drift and periods of\nanomalously low or high variability improved when we applied replaced anomalous\nmeasurements with forecasts, but this inflated false positive rates.\nFeature-based methods also performed well on high priority anomalies, but were\nalso less proficient at detecting lower priority anomalies, resulting in high\nfalse negative rates. Unlike regression-based methods, all feature-based\nmethods produced low false positive rates, but did not and require training or\noptimization. Rule-based methods successfully detected impossible values and\nmissing observations. Thus, we recommend using a combination of methods to\nimprove anomaly detection performance, whilst minimizing false detection rates.\nFurthermore, our framework emphasizes the importance of communication between\nend-users and analysts for optimal outcomes with respect to both detection\nperformance and end-user needs. Our framework is applicable to other types of\nhigh frequency time-series data and anomaly detection applications.\n
BibTeX
@inproceedings{leigh2019framework,
  author = {Catherine Leigh and Omar Alsibai and Rob J. Hyndman and Sevvandi Kandanaarachchi and Olivia C. King and James McGree and Catherine Neelamraju and Jennifer Strauss and Priyanga Dilini Talagala and Ryan S. Turner and Kerrie Mengersen and Erin E. Peterson},
  title = {{A framework for automated anomaly detection in high frequency\textbackslash\{\}n water-quality data from in situ sensors}},
  booktitle = {The Science of The Total Environment},
  year = {2019},
  doi = {10.1016/j.scitotenv.2019.02.085},
}
Figure