A feature-based framework for detecting technical outliers in water-quality data from in situ sensors

Priyanga Dilini Talagala, Rob J. Hyndman, Catherine Leigh, Kerrie Mengersen, Kate Smith‐Miles

Abstract

Outliers due to technical errors in water-quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. However, outlier detection through manual monitoring is unfeasible given the volume and velocity of data the sensors produce. Here, we proposed an automated framework that provides early detection of outliers in water-quality data from in situ sensors caused by technical issues. The framework was used first to identify the data features that differentiate outlying instances from typical behaviours. Then statistical transformations were applied to make the outlying instances stand out in transformed data space. Unsupervised outlier scoring techniques were then applied to the transformed data space and an approach based on extreme value theory was used to calculate a threshold for each potential outlier. Using two data sets obtained from in situ sensors in rivers flowing into the Great Barrier Reef lagoon, Australia, we showed that the proposed framework successfully identified outliers involving abrupt changes in turbidity, conductivity and river level, including sudden spikes, sudden isolated drops and level shifts, while maintaining very low false detection rates. We implemented this framework in the open source R package oddwater.

Introduction

In situ sensors are the primary tools for modern environmental monitoring, providing high-frequency data essential for understanding river health. However, these sensors are prone to "technical outliers"—errors caused by hardware glitches, battery failure, or biofouling—rather than actual environmental events like floods or storms. Distinguishing these errors is critical; if technical outliers are not identified, subsequent environmental models are built on unreliable data.

Manual data cleaning is impossible given the sheer volume and velocity of modern sensor networks. Furthermore, most automated tools are "supervised," requiring massive hand-labeled datasets. This paper proposes an unsupervised solution that identifies technical errors without needing prior examples of every possible failure mode.

Methodology

The researchers developed an automated, six-step framework designed to catch technical glitches in real-time. This framework is implemented in the open-source R package oddwater.

The proposed automated framework for outlier detection in water-quality data.

The core innovation of this framework is its feature-based approach. Instead of analyzing raw sensor readings, the system applies statistical transformations to highlight specific data features that differentiate errors from typical behavior. These transformations include first derivatives to handle irregular time series, one-sided derivatives to separate spikes from typical upward trends, and relative differences to detect sudden movements.

Summary of statistical transformations and data features used to highlight outlying instances.

By mapping these transformed variables, the problem shifts from a temporal context (time series) to a spatial context (points in a high-dimensional space). Unsupervised algorithms, such as K-Nearest Neighbors (KNN), then calculate "outlierness" scores based on how isolated a point is from its neighbors. To avoid arbitrary cutoffs, the framework uses Extreme Value Theory (EVT) to calculate a probabilistic threshold, providing a specific level of statistical confidence for each flagged outlier.

Datasets and Evaluation

The framework was evaluated using real-world data from two sites in Australia's Mackay-Whitsunday region: Sandy Creek and the Pioneer River. Sensors measured turbidity, conductivity, and river level. Experts manually labeled these datasets to provide a "ground truth" for performance evaluation.

Key results include:

  • Top Performance: The combination of one-sided derivative transformations and KNN-based algorithms (specifically KNN-AGG and KNN-SUM) yielded the highest accuracy.
  • High Reliability: At Sandy Creek, the best model achieved an Optimized Precision (OP) of 0.8329 and a Negative Predictive Value (NPV) of 0.9996, meaning it was exceptionally reliable at identifying typical points while catching the vast majority of technical errors.
  • Computational Efficiency: Algorithms like HDoutliers and KNN-SUM processed thousands of points in under 230 milliseconds, making them ideal for near-real-time monitoring.

Discussion and Future Work

A significant insight from the study is the power of the one-sided derivative. Because natural environmental processes are often slow while technical errors are abrupt, this transformation is highly effective at separating sudden sensor spikes from the gradual decay seen in real events. Furthermore, by treating time series data as points in a high-dimensional space, the researchers could leverage powerful distance-based algorithms usually reserved for static datasets.

While the current framework is optimized for abrupt changes (spikes and level shifts), it may struggle with slow-moving "drifts" caused by gradual biofouling. Future work will explore expanding the framework to account for spatio-temporal correlations—checking if a sensor's reading makes sense relative to neighboring sensors upstream and downstream—and incorporating residual analysis to catch more complex anomaly types.

Summary updated

Abstract

Outliers due to technical errors in water-quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. However, outlier detection through manual monitoring is unfeasible given the volume and velocity of data the sensors produce. Here, we proposed an automated framework that provides early detection of outliers in water-quality data from in situ sensors caused by technical issues.The framework was used first to identify the data features that differentiate outlying instances from typical behaviours. Then statistical transformations were applied to make the outlying instances stand out in transformed data space. Unsupervised outlier scoring techniques were then applied to the transformed data space and an approach based on extreme value theory was used to calculate a threshold for each potential outlier. Using two data sets obtained from in situ sensors in rivers flowing into the Great Barrier Reef lagoon, Australia, we showed that the proposed framework successfully identified outliers involving abrupt changes in turbidity, conductivity and river level, including sudden spikes, sudden isolated drops and level shifts, while maintaining very low false detection rates. We implemented this framework in the open source R package oddwater.
BibTeX
@inproceedings{talagala2019feature,
  author = {Priyanga Dilini Talagala and Rob J. Hyndman and Catherine Leigh and Kerrie Mengersen and Kate Smith‐Miles},
  title = {{A feature-based framework for detecting technical outliers in water-quality data from in situ sensors}},
  booktitle = {Water},
  year = {2019},
  doi = {10.1029/2019wr024906},
}
Figure