Detecting energy theft with partially observed anomalies

Hua Chen, Rui Ma, Xiufeng Liu, Ruyu LiuInternational Journal of Electrical Power & Energy Systems 2024 · DOI

TL;DR

This paper introduces a novel method for detecting energy theft in power grids by effectively combining feature extraction, fuzzy clustering, and ensemble learning. It tackles the challenge of limited labeled data and the emergence of new theft patterns, achieving high accuracy with significantly lower computational costs than deep learning models.

What Problem Does This Paper Address?

Energy theft is a major issue for power companies, leading to financial losses and grid instability. Traditional detection methods often struggle because they rely on ample labeled data and can't easily adapt to new, unseen theft techniques. This is exacerbated by "partially observed anomalies," where only a fraction of theft data is labeled, and some types of theft are completely unknown. The authors aim to develop a robust and efficient system that can detect energy theft even with incomplete information and evolving patterns.

Methods

The proposed method is a three-pronged approach:

  1. Feature Extraction: They use Discrete Wavelet Transform (DWT) to break down energy consumption data into different frequency components. This allows them to capture both short-term fluctuations and long-term trends. From these components, they extract eight key statistical and spectral features (mean, standard deviation, skewness, kurtosis, energy, entropy, peak, and peak-to-average ratio) per decomposition level. These features are then normalized.
  2. Fuzzy Clustering: Fuzzy C-Means (FCM) is employed to group observed anomalies into distinct classes. Unlike traditional k-means, FCM allows data points to belong to multiple clusters with varying degrees of membership, which is crucial for handling the inherent uncertainty and overlap in energy theft patterns. This step also helps filter unlabeled data by identifying potential anomalies and reliable normal data points based on their proximity to anomaly clusters.
  3. Ensemble Learning: A weighted multi-class logistic regression model is used. For each identified anomaly cluster, a separate logistic regression model is trained. These models are weighted based on the confidence of the data points and the importance of each base learner. This ensemble approach allows for effective differentiation between various theft types and normal consumption patterns, even with pseudo-labeled data from the clustering step.

Datasets

The authors used a synthetic dataset for their experiments. They combined normal load profiles from the Irish Smart Energy Trial (half-hourly data from over 5000 homes and businesses) with artificially generated energy theft patterns. Six different functions were used to simulate various theft behaviors, resulting in five distinct energy theft classes and normal consumption patterns. This synthetic approach allowed them to control anomaly types and data characteristics, which is often difficult with real-world energy theft data due to privacy concerns.

Key Insights and Ideas

  • Leveraging the Unknown: The core innovation is the ability to effectively use unlabeled data and generalize to unobserved anomaly classes, a significant challenge in anomaly detection.
  • Fuzzy Logic for Ambiguity: Using fuzzy clustering is key to handling the inherent ambiguity and overlap in energy theft patterns, which rigid clustering methods might miss.
  • Ensemble Power: Combining multiple specialized logistic regression models (one per anomaly type) provides a more nuanced and accurate detection system than a single, general model.
  • Feature Importance Matters: A systematic feature analysis revealed that the "peak-to-average ratio" and "entropy" are particularly crucial for detecting energy theft, guiding future feature selection.

Key Results

  • High Accuracy: The proposed method achieved superior performance compared to several baseline methods (LR, PU, SS, KM, iForest) and deep learning models (LSTM, VAE) across various energy theft classes.
  • AUC-PR and AUC-ROC: The method consistently outperformed baselines, with average AUC-PR of 0.76 and AUC-ROC of 0.84. While deep learning models were slightly higher on average, the proposed method was competitive.
  • Computational Efficiency: The method required significantly less computational time (under 10 minutes) compared to deep learning models, making it more practical for real-world deployment.
  • Robustness: The method demonstrated robustness to increasing noise injection ratios and sensitivity analysis showed that optimal parameters for DWT decomposition and fuzzy clustering could be identified for different theft types.

Limitations and Future Work

  • Data Dependency: The performance relies on the accuracy and representativeness of labeled anomalies. Evolving theft patterns might require continuous model updates.
  • Hyperparameter Tuning: The method has several hyperparameters (e.g., number of clusters, fuzziness degree) that need careful tuning for specific datasets.
  • Generalization: While effective for energy theft, the method's applicability to other anomaly detection domains (intrusion, fraud) needs further investigation.

Future work could explore:

  • More advanced feature extraction techniques (e.g., wavelet packet transform, deep learning-based methods).
  • Alternative clustering algorithms (e.g., possibilistic c-means, evidential c-means).
  • More sophisticated ensemble learning strategies (bagging, boosting, stacking).
  • Distributed and parallel processing for scalability.
  • Incorporating domain knowledge and expert feedback.

Summary updated

Abstract

Energy theft poses a significant threat to the power industry, causing financial losses and grid instability. Existing detection methods often struggle with limited labeled data and the emergence of new, unobserved theft patterns. To address these challenges, we propose a novel method for energy theft detection that effectively leverages both partially observed anomalies and unlabeled data. Our approach integrates Discrete Wavelet Transform (DWT) for feature extraction, Fuzzy C-Means clustering for anomaly grouping, and weighted multi-class logistic regression for ensemble learning. Extensive experiments on a realistic dataset demonstrate that our method achieves high detection accuracy, outperforming several state-of-the-art methods, including deep learning models, while maintaining significantly lower computational cost. This robust and efficient approach enables effective detection of unobserved anomaly classes and reduces false positives, making it a valuable tool for developing reliable energy theft detection systems. We further conduct a feature importance analysis to identify influential features for optimizing detection accuracy and efficiency. • Novel energy theft detection method integrating feature extraction, fuzzy clustering, and ensemble learning. • Systematic feature importance analysis identifying key indicators for theft detection. • Extensive performance comparison with state-of-the-art methods, demonstrating superior accuracy. • Robust against noise, effective across various energy theft scenarios with different data sets.
BibTeX
@article{chen2024detecting,
  author = {Hua Chen and Rui Ma and Xiufeng Liu and Ruyu Liu},
  title = {{Detecting energy theft with partially observed anomalies}},
  journal = {International Journal of Electrical Power \& Energy Systems},
  year = {2024},
  doi = {10.1016/j.ijepes.2024.110323},
}
Figure