TL;DR
This paper tackles the critical issue of low-quality monitoring data in hydropower units, characterized by anomalies and missing values. The authors propose a novel two-stage approach: first, using HDBSCAN for robust anomaly detection and data cleaning, and second, employing a Wasserstein Slim Generative Adversarial Imputation Network with Gradient Penalty (WSGAIN-GP) to intelligently fill in missing data. This combined method significantly enhances data quality, paving the way for more reliable condition monitoring and prediction in hydropower systems.
What Problem Does This Paper Address?
Hydropower units are vital components of modern power systems, and their accurate condition monitoring is paramount for reliable operation and prediction. However, the data collected from these units often suffers from quality issues, primarily data anomalies (outliers, incorrect readings) and missing values. These problems arise from various sources, including interference, equipment malfunctions, and data acquisition/transmission link failures. Traditional methods for handling these issues often fall short, especially when dealing with the complex, varying distributions and temporal dependencies inherent in hydropower unit data. Existing clustering methods struggle with varying densities, and simple imputation techniques fail to capture intricate data patterns. This gap hinders the effectiveness of downstream tasks like fault diagnosis, trend prediction, and state warning, necessitating a more sophisticated approach to data quality enhancement.
Methods
The proposed methodology is a two-pronged attack on data quality issues:
- Anomaly Detection and Cleaning (HDBSCAN): The authors leverage the Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) algorithm. Unlike traditional DBSCAN, HDBSCAN is adept at handling datasets with varying densities and automatically identifies noise points (outliers) without requiring extensive parameter tuning. It works by building a hierarchical structure of clusters and then pruning it based on a minimum cluster size, effectively separating dense regions from sparse ones. This allows for adaptive detection and removal of anomalous data points, resulting in a cleaner dataset.
- Data Imputation (WSGAIN-GP): For filling in the missing values, the paper introduces a Wasserstein Slim Generative Adversarial Imputation Network with Gradient Penalty (WSGAIN-GP). This is an advanced generative adversarial network (GAN) architecture.
- GAN Foundation: GANs consist of a generator (G) that creates synthetic data and a discriminator (D) that tries to distinguish between real and generated data.
- WSGAIN Improvement: The WSGAIN variant builds upon previous GAN architectures (like GAIN and SGAIN) by incorporating the Wasserstein distance to measure the difference between real and generated data. This improves training stability and prevents issues like mode collapse.
- Gradient Penalty (GP): The WSGAIN-GP further enhances stability and performance by adding a gradient penalty to the loss function, which is a more robust alternative to weight clipping used in earlier WGANs.
- Imputation Focus: The WSGAIN-GP is specifically designed for data imputation. It learns the underlying data distribution and temporal dependencies from the available data and uses this knowledge to generate realistic values for the missing entries.
Datasets
The study utilizes an actual operational monitoring dataset from a mixed-flow hydropower unit at the Fengtan hydropower station in the Central China region. This dataset includes various state parameters such as vibration, deflection, pressure pulsation, air gap, stator temperature, oil level, active power, water level, and rotational speed. For the analysis, specific parameters like upper guide swing deflection (S), head (H), and active power (P) were selected, forming a three-dimensional dataset. The paper also mentions using a short-term complete state sequence dataset for comparative analysis of different imputation methods at various missing rates.
Key Insights and Ideas
- HDBSCAN's Adaptability: The paper highlights HDBSCAN's superiority in identifying anomalies in complex, multi-dimensional datasets with varying densities, outperforming traditional methods like DBSCAN, OPTICS, LOF, and K-Means in terms of clustering effectiveness (measured by SCI).
- WSGAIN-GP's Fidelity: The WSGAIN-GP model demonstrates a remarkable ability to learn and replicate the distribution and temporal characteristics of real hydropower data, resulting in imputed values that closely match the original measurements.
- Synergy of Anomaly Detection and Imputation: The sequential application of HDBSCAN for cleaning followed by WSGAIN-GP for imputation creates a powerful pipeline that addresses both major data quality issues effectively.
- Robustness to Missing Data: The WSGAIN-GP model shows consistent performance and superior accuracy across a range of missing data rates, indicating its generalization capability.
Key Results
- Anomaly Detection: The HDBSCAN model achieved a clustering contour coefficient (SCI) of 0.4935 for the upper guide Y-axis swing data, which was higher than other comparative methods.
- Data Imputation Accuracy: The KL divergence, JS divergence, and Hellinger distance between the imputed and original data distributions were close to zero, indicating high accuracy.
- Comparative Imputation Performance: The WSGAIN-GP model consistently achieved the lowest Root Mean Square Error (RMSE) compared to SGAIN, GAIN, and KNN across various missing rates (10%, 30%, 50%, 70%). For instance, in the upper guide X-axis swing imputation, the average RMSE reduction was 0.0936.
Limitations and Future Work
- Computational Cost: While not explicitly detailed as a limitation, GAN-based models, including WSGAIN-GP, are known for their computational intensity during training.
- Data Availability: The authors state that the raw/processed data cannot be shared at this time due to participant privacy concerns.
- Future Work: The paper suggests that the robust data quality enhancement provided by this method lays a strong foundation for subsequent trend prediction and state warning systems in hydropower units. Further research could explore the application of this combined approach to other types of industrial monitoring data.
Summary updated