TL;DR
This paper introduces a comprehensive, ten-step framework for automatically detecting anomalies in high-frequency water quality data from in-situ sensors. This is crucial because manual inspection of the massive data streams generated by these sensors is infeasible, and anomalies can lead to incorrect environmental assessments. The framework evaluates various anomaly detection methods, offering a structured approach for selecting the most suitable ones for specific end-user needs.
What Problem Does This Paper Address?
Monitoring water quality in rivers is increasingly done using automated in-situ sensors, which provide continuous data streams. However, these sensors are prone to technical issues (e.g., sensor drift, calibration errors, power failures) that generate anomalous data. These anomalies can be mistaken for genuine environmental changes, leading to flawed water quality assessments and policy decisions. Manually sifting through the high-volume, high-velocity data to identify these anomalies is practically impossible. Therefore, there's a critical need for automated anomaly detection (AD) methods that can reliably flag these technical errors in near-real time.
Methods
The paper proposes a ten-step Anomaly Detection (AD) Framework. Key steps include:
- Identifying end-user needs and goals: Understanding what the user wants to achieve with the data (e.g., visualization, quality control, forecasting) is paramount for prioritizing anomaly types.
- Defining and ranking anomalies: Anomalies are categorized (e.g., sudden spikes, drift, missing values) and ranked by importance based on their potential impact if undetected.
- Selecting suitable AD methods: The framework evaluates several AD approaches, including:
- Automated classification rules: Simple "if-then" rules for detecting obvious anomalies like impossible values or out-of-range readings.
- Regression-based methods: These models learn "normal" behavior from historical data and flag deviations. This includes naïve prediction, linear autoregression, ARIMA, and multivariate regression with ARIMA errors (RegARIMA). The paper also explores the Anomaly Detection and Mitigation (ADAM) approach, which uses forecasts instead of actual anomalous observations to predict subsequent data points.
- Feature-based methods: These methods transform data and then use techniques like k-nearest neighbors (kNN) or HDoutliers to identify anomalies based on distance or density in a high-dimensional feature space. These are unsupervised and data-driven.
- Evaluation and comparison: Standard metrics like accuracy, error rate, positive predictive value (PPV), and negative predictive value (NPV) are used, along with computation time.
Datasets
The study utilizes high-frequency water quality data collected from in-situ sensors in two rivers flowing into the Great Barrier Reef lagoon in northeast Australia:
- Pioneer River (PR)
- Sandy Creek (SC)
The data collected includes:
- Turbidity (NTU)
- Electrical Conductivity (µS/cm)
- River Level (m)
The time series data spans from March 12, 2017, to March 12, 2018.
Key Insights and Ideas
- End-user needs drive anomaly detection: The importance of detecting different anomaly types is not universal but depends heavily on the end-user's goals. For visualization, sudden spikes are high priority, while for sensor maintenance, drift might be more critical.
- No single method is best for all anomalies: Different AD methods excel at detecting different types of anomalies. Rule-based methods are good for simple errors, regression-based methods for detecting deviations from learned patterns, and feature-based methods for complex multivariate relationships.
- ADAM can reduce false positives but has trade-offs: While the ADAM approach (using forecasts to mitigate anomalies) can reduce false positives in some cases, it can also lead to inflated false positive rates in others, particularly when dealing with complex data patterns. However, it proved useful for detecting certain Class 3 anomalies that AD alone missed.
- Multivariate relationships can complicate detection: Incorporating multiple water quality variables as covariates (in RegARIMA) did not always improve performance and could even degrade it, especially when relationships between variables were complex or non-stationary.
- Framework transferability: The proposed ten-step framework is designed to be transferable to other types of high-frequency time-series data and anomaly detection applications beyond water quality.
Key Results
- Automated classification rules successfully detected all Class 2 anomalies (impossible values, out-of-sensor-range, missing data) with no false positives.
- Regression-based methods generally showed high accuracy and low error rates for detecting high-priority anomalies (Class 1: spikes, shifts). However, the ADAM approach often resulted in high false positive rates.
- Feature-based methods performed well on high-priority anomalies but struggled with lower-priority ones, leading to high false negative rates.
- ARIMA models were found to be a useful regression-based method, especially when covariates were unavailable or problematic.
- RegARIMA did not consistently outperform simpler ARIMA models, and in some cases, including additional covariates negatively impacted performance.
- ADAM was particularly useful for detecting Class 3 anomalies (drift, high variability) that AD alone missed, though it increased false positives.
Limitations and Future Work
- Overfitting: Regression models may have been over-fitted due to training and testing on the same data (minus anomalies). Future work could explore using data from nearby sites or different time periods for training.
- Nonstationarity: Diagnostic plots revealed evidence of nonstationarity (changing variance and means) in the residuals of regression-based methods, which could affect the accuracy of prediction intervals.
- Data imbalance: The significant imbalance between normal and anomalous observations in the datasets posed challenges for interpreting performance metrics.
- Feature-based method limitations: Feature-based methods, while efficient, were not proficient at detecting Class 3 anomalies, as they were primarily designed for Class 1 anomalies. Future work could explore other transformations or methods to better target Class 3 anomalies.
- Event-based cross-validation: The authors suggest exploring event-based or walk-forward cross-validation techniques for future research to address issues related to time-specific behaviors and limited data length.
- Differentiating regimes: For highly seasonal rivers, differentiating between flow regimes for training purposes could improve regression-based AD performance. Techniques like discrete-space hidden Markov models could be used for this segmentation.
Summary updated