TL;DR
This paper introduces a novel, sliding-window-based data quality control (DQC) method for seafloor observatories, specifically designed to detect and repair outliers and missing data in time-series oceanographic measurements. By combining statistical analysis with expert knowledge and leveraging an improved ARIMA model, the method effectively ensures the reliability of data crucial for marine scientific research.
What Problem Does This Paper Address?
Seafloor observatories are increasingly vital for long-term, continuous, high-resolution oceanographic data collection. However, the sheer volume and continuous nature of this data present significant challenges for ensuring its quality. Sensors can degrade over time, leading to issues like outliers (erroneous data points) and data gaps (missing values). These problems can severely compromise the continuity and scientific value of the data, making robust data quality control essential. Existing methods often focus solely on detection or require extensive manual intervention. This paper addresses the need for an automated, integrated approach that not only identifies but also repairs these data quality issues in seafloor observatory data.
Methods
The proposed method is a three-part system applied using a sliding window approach:
- General Pretest: This initial stage performs quick checks to identify and handle redundant data points, flag "stuck" values (where a sensor output remains constant for too long), and detect data gaps. It also acts as a router, directing data points to subsequent processing steps or directly to interpolation if they are identified as missing or stuck.
- Data Outlier Detection: Two methods are employed:
- Range Rationality Test: This uses domain expert-defined upper and lower limits to flag data points that fall outside expected physical or sensor ranges.
- ARIMA-Based Test: An improved Autoregressive Integrated Moving Average (ARIMA) model, adapted for time-series data with a sliding window and cleaned input data, generates a predicted value. Outliers are identified by comparing the predicted value to the observed value; a significant relative error flags a point as suspect.
- Data Interpolation: For flagged outliers and missing data, the sliding-window ARIMA model is used to generate replacement values. This approach leverages the model's forecasting accuracy to provide reasonable estimates for missing or erroneous data.
A Quality Control Flag System is also introduced to provide metadata about the data's quality status and the processing steps applied.
Datasets
The method was tested and verified using real observed data from the Xiaoqushan Seafloor Observatory in the East China Sea. Specifically, CTD (Conductivity, Temperature, Depth) and pH measurements collected between August 2013 and September 2014 were used. The data was divided into subsets for evaluation, with manual labeling of outliers by domain experts used for comparison.
Key Insights and Ideas
- Repair is as Crucial as Detection: The paper argues that true data quality control involves not just identifying bad data but also providing reasonable replacements, thus preserving data continuity and scientific utility.
- ARIMA for Oceanographic Time Series: Adapting and improving the ARIMA model, known for time-series forecasting, proves effective for both detecting anomalies and interpolating missing values in the complex, often non-stationary data from seafloor observatories.
- Sliding Window for Adaptability: The sliding window approach allows the ARIMA model to continuously adapt to changing data characteristics, improving prediction accuracy and making the method suitable for real-time applications.
- Hybrid Approach is Key: Combining statistical methods (like range tests and ARIMA) with domain expert knowledge (for setting limits and validating results) leads to a more robust and reliable DQC system.
Key Results
- Outlier Detection: The proposed ARIMA-based outlier detection method achieved an F1 score of 0.9506 for pH data, outperforming traditional methods like the 3-standard deviation (3sd) and 2-standard deviation (2sd) methods. The ARIMA method demonstrated superior precision and recall, effectively balancing the detection of true outliers with minimizing false positives.
- Data Interpolation: The ARIMA model achieved low error rates for both single-point and successive multipoint interpolation across CTD measurements (temperature, conductivity, pressure). For example, the Mean Absolute Percentage Error (MAPE) for single-point interpolation ranged from 0.0015% to 0.0226%, and for successive multipoint interpolation, it ranged from 0.0241% to 0.0973%. These low error values indicate high accuracy in repairing data gaps and outliers.
- Efficiency: The method is described as having affordable computing complexity and timely response, making it suitable for real-time observed data.
Limitations and Future Work
- Adaptive Thresholds: The authors note that the relative error threshold used in the ARIMA-based outlier detection currently needs to be pre-defined for each measurement. Future work could involve developing adaptive thresholds.
- Unsupervised Methods: Exploring unsupervised methods could further enhance the robustness and adaptability of outlier detection in seafloor observatory data.
- System Development: The authors plan to develop a comprehensive seafloor observatory DQC method and framework, including a graphical user interface (GUI), to facilitate broader application.
Summary updated