TL;DR
This paper introduces a robust framework using Generalized Additive Models (GAMs) and Autoregressive Integrated Moving Average (ARIMA) models to reconstruct missing high-frequency water quality data from in-situ sensors. Their method effectively fills data gaps, even for complex events, improving the reliability of environmental monitoring and management.
What Problem Does This Paper Address?
High-frequency in-situ sensors are revolutionizing environmental monitoring, providing unprecedented insights into aquatic ecosystems. However, these sensors are susceptible to technical issues like biofouling and power failures, leading to data gaps and anomalous values. These missing data points can significantly skew analyses, leading to inaccurate trend detection, biased statistics, and reduced statistical power. Traditional methods for handling missing data were often developed for lower-frequency datasets and may not be suitable for the dense, dynamic nature of high-frequency sensor streams. This paper addresses the critical need for reliable methods to reconstruct these missing high-frequency water quality data, specifically focusing on nitrate concentration in rivers, to ensure the integrity of environmental monitoring and decision-making.
Methods
The authors propose a two-tiered reconstruction framework:
- Generalized Additive Models (GAMs): When all relevant covariates (other water quality parameters like temperature, turbidity, dissolved oxygen, etc.) are available at the same time step as the missing data point, a GAM is employed. This model uses smooth functions of the covariates to predict the missing value. The selection of the "best" GAM model is guided by the Akaike Information Criterion (AIC).
- Autoregressive Integrated Moving Average (ARIMA) Models: If one or more covariates are missing at a given time step, the GAM approach cannot be used. In such cases, an ARIMA model is applied. This model uses the 500 preceding observations of the target variable (nitrate concentration) to predict the missing value.
The framework also incorporates the concept of prediction intervals (95%) to quantify the uncertainty associated with the reconstructed values.
Datasets
The primary dataset used for both developing and evaluating the reconstruction method is a two-year time series of high-frequency nitrate concentration data collected from the Arikaree River in eastern Colorado, USA. This dataset was sourced from the National Ecological Observatory Network (NEON).
In addition to nitrate concentration, the dataset includes synoptically collected data for:
- Water temperature
- Turbidity
- Conductance
- Elevation
- Dissolved oxygen
These additional variables serve as covariates in the GAM models. The paper also describes a simulation study where data were artificially removed from this real-world dataset to systematically test the reconstruction method under various missing data scenarios.
Key Insights and Ideas
- Covariates are King: The availability of co-located, high-frequency sensor data for other water quality parameters (covariates) significantly improves the accuracy of nitrate data reconstruction. This highlights the value of deploying multi-parameter sensor suites.
- GAMs Shine with Data: When covariates are available, GAMs perform exceptionally well, accurately capturing even sudden, event-based peaks in nitrate concentration. This suggests that understanding the relationships between different water quality parameters is crucial for robust data imputation.
- ARIMA as a Fallback: ARIMA models provide a necessary fallback when covariates are unavailable, but their performance is generally lower, especially when dealing with longer sequences of missing data.
- Sequences are Tougher: Reconstructing contiguous sequences of missing data is more challenging than filling individual, sporadically missing points. The longer the gap, the less precise the reconstruction tends to be.
- Beyond Point Data: The study emphasizes the importance of addressing not just isolated missing points but also longer periods of data loss, which can occur due to sensor malfunctions or quality control procedures.
Key Results
- Sporadic Data: For sporadically missing point data, the method achieved good accuracy. In 72% of cases, predicted values fell within the sensor's precision interval.
- Event Reconstruction: The method successfully reconstructed sudden, event-based peaks in nitrate concentration when covariates were available.
- Contiguous Gaps: Performance declined for contiguous missing data. For a single day of missing data, the daily pattern was not well reconstructed. For a three-month gap, some extremely high nitrate concentrations were predicted based on covariates, which might not have been flagged by anomaly detection.
- Simulation Performance: In simulations, even with 40% of point data removed, the method recovered approximately 70% of the data within the sensor's precision interval.
- Prediction Interval: The median prediction interval for the reconstructed data was low (1.34 µmol/L), indicating that many reconstructed values were more precise than the sensor's accuracy.
Limitations and Future Work
- Covariate Dependency: The performance of the reconstruction method is heavily dependent on the availability and quality of covariates. When covariates are missing, the reliance on ARIMA models leads to reduced accuracy.
- Sequence Length: The accuracy of reconstruction decreases with the length of contiguous missing data sequences, particularly for week-long gaps.
- Generalizability: While the authors believe the general methodology is transferable to other sites and water quality variables, they acknowledge that specific relationships between variables may differ.
- Future Directions:
- Develop a more flexible framework where ARIMA is only used when no covariate data is available, rather than when any covariate is missing.
- Explore other time series modeling techniques, such as Seasonal ARIMA (SARIMA) for seasonal data or deep learning methods like Long Short-Term Memory (LSTM) networks.
- Conduct further research to confirm the applicability of the method across different sites and environmental data types to generalize the framework.
- Investigate the relationships between various water quality variables to optimize the deployment of co-located sensors for improved data reconstruction.
Summary updated