Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters

Shuxian Sun, Chunyu Liu, Yiqun Zhu, Haihang He, Shuai Xiao, Jiabao Wen

TL;DR

This paper introduces a novel Deep Reinforcement Learning (DRL) approach for detecting abnormal data in smart meters, a critical task for grid security and efficiency. By combining a Deep Q-Network (DQN) with Fuzzy C-Means (FCM) for state prediction, the model achieves improved accuracy and efficiency in identifying anomalies compared to existing methods.

What Problem Does This Paper Address?

The proliferation of smart meters in modern power grids generates massive amounts of data, posing significant challenges for security management and anomaly detection. Abnormal readings from smart meters can lead to financial losses for both power companies and consumers, and can disrupt grid operations. Traditional methods for detecting these anomalies often struggle with the scale, diversity, and complexity of the data, or require extensive manual intervention. This paper aims to address this gap by developing an intelligent, automated system for detecting abnormal smart meter data.

Methods

The core of this paper's contribution lies in its proposed Deep Reinforcement Learning (DRL) model for abnormal data detection. The approach leverages a Deep Q-Network (DQN), which combines the decision-making capabilities of Reinforcement Learning (RL) with the perception abilities of Deep Learning (DL).

Key methodological components include:

  • Deep Q-Network (DQN): This forms the backbone of the detection system. It consists of two main networks:
  • Main Network (MainNet): Responsible for predicting Q-value estimations.
  • Target Network (TargetNet): Used for predicting the "true" Q-values, providing a stable target for learning.
  • Q-Learning Algorithm: The underlying RL algorithm used to train the DQN. It aims to learn an optimal policy by maximizing expected future rewards.
  • Fuzzy C-Means (FCM): A novel integration into the DQN framework. FCM is used to predict the state information for the next time step. This is a key innovation, as it improves the computational accuracy of the DRL model by providing a more robust prediction of future states, especially in the presence of noisy data.
  • Experience Replay: A standard DQN technique used to store past experiences (state, action, reward, next state) and sample from them during training. This helps to break correlations between consecutive samples and improve learning stability.
  • Max-Min Scaling: A data normalization technique applied to the dataset to scale all values to a range between 0 and 1, ensuring that features with different scales do not disproportionately influence the model.

Datasets

The study utilized a real-world electricity consumption dataset from the State Grid Corporation of China (SGCC). This dataset comprised 149,186 samples collected between January 2014 and February 2017, including both normal and abnormal consumption patterns. The dataset was randomly split into training (60%), validation (20%), and test (20%) sets.

Key Insights and Ideas

  • FCM for State Prediction is Crucial: The integration of Fuzzy C-Means for predicting future states significantly enhances the accuracy of the DRL model. This suggests that explicitly modeling uncertainty and fuzzy boundaries in state transitions can be highly beneficial for anomaly detection in time-series data.
  • DRL Outperforms Traditional Methods: The proposed DRL approach demonstrates superior performance compared to traditional methods like Linear Regression (LR), Support Vector Machines (SVM), and even other deep learning models like CNN, LSTM, and TCN, particularly in terms of accuracy and robustness against unknown attacks.
  • Efficiency Gains: The proposed method not only achieves high accuracy but also exhibits a shorter detection time, especially as the number of users increases. This is attributed to the model's strong feature aggregation capabilities.
  • Epochs and Batch Size Matter: The experimental analysis highlights the importance of selecting appropriate hyperparameters. The optimal epoch count for detection performance was found to be around 60, suggesting that excessive training can lead to overfitting. Similarly, batch size influences convergence speed and performance, with smaller batch sizes potentially achieving good AUC with fewer epochs.

Key Results

The proposed Deep Reinforcement Learning model (referred to as FRL in the paper) achieved the following key results on the SGCC dataset:

  • Accuracy (ACC): 0.947
  • Area Under the Curve (AUC): 0.827

These results represent a significant improvement over existing methods, including DRL [26] (ACC: 0.823, AUC: 0.690), RL [27] (ACC: 0.856, AUC: 0.724), HDFC [28] (ACC: 0.912, AUC: 0.728), and TAD [29] (ACC: 0.914, AUC: 0.746). The paper also shows that the FRL model has better performance than traditional LR and SVM methods, and competitive performance with CNN, LSTM, and TCN, while being less dependent on extensive training data.

Limitations and Future Work

  • Limitations:
    • The paper notes that while deep neural network methods offer better performance, they require a large amount of training data. The proposed method is less dependent on training data for state prediction, but this general observation about deep learning models still holds.
  • The study focuses on detecting abnormal data in smart meters. While it mentions "unknown attacks," further investigation into specific types of adversarial attacks and their resilience would be beneficial.
  • Future Work:
    • The authors suggest that further research into the optimal selection of hyperparameters (embedding dimension, number of layers, dropout probability) can lead to even better performance.
  • The paper implies that the model's ability to extract nonlinear features is strong, suggesting potential for further exploration in complex power grid scenarios.
  • While the paper demonstrates high stability and reliability, real-world deployment and continuous monitoring would be valuable for further validation.

Summary updated

Abstract

The rapidly growing power data in smart grids have created difficulties in security management. The processing of large-scale power data with the use of artificial intelligence methods has become a hotspot research topic. Considering the early warning detection problem of smart meters, this paper proposes an abnormal data detection network based on Deep Reinforcement Learning, which includes a main network and a target network composed of deep learning networks. This work uses the greedy policy algorithm to find the action of the maximum value of Q based on the Q-learning method to obtain the optimal calculation policy. It also uses the reward value and discount factor to optimize the target value. In particular, this study uses the fuzzy c-means method to predict the future state information value, which improves the computational accuracy of the Deep Reinforcement Learning model. The experimental results show that compared with the traditional smart meter data anomaly detection method, the proposed model improves the accuracy of meter data anomaly detection.
BibTeX
@inproceedings{sun2022deep,
  author = {Shuxian Sun and Chunyu Liu and Yiqun Zhu and Haihang He and Shuai Xiao and Jiabao Wen},
  title = {{Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=mrSexMfZ6nl},
}

Loading related content...

Figure