Online Anomaly Energy Consumption Detection Using Lambda Architecture

Xiufeng Liu, Xiufeng Liu 0001, Nadeem Iftikhar, Per Sieverts Nielsen, Alfred HellerDawak 2016 · DOI

TL;DR

This paper tackles the challenge of detecting anomalous energy consumption in real-time by proposing a novel system that leverages the Lambda architecture. By combining a statistical anomaly detection algorithm with a robust big data processing framework, the system efficiently updates detection models and provides timely alerts for unusual energy usage.

What Problem Does This Paper Address?

The proliferation of smart meters has generated vast amounts of energy consumption data, creating a significant opportunity to identify unusual patterns. These anomalies can signal various issues, from user behavior (e.g., forgetting to turn off appliances) to utility-level problems (e.g., energy leakage or theft). Effectively detecting these anomalies in real-time is crucial for timely intervention and optimization. Traditional data mining approaches often struggle with the sheer volume and velocity of this data, requiring offline model training that can lead to outdated detection capabilities. The paper aims to bridge this gap by enabling both efficient model updates and real-time anomaly detection on a large scale.

Methods

The core of the proposed solution is the Lambda architecture, a system design pattern that combines batch and stream processing to achieve both low-latency (near real-time) and high-throughput data processing.

  • Anomaly Detection Algorithm: The paper proposes a statistical anomaly detection method based on a periodic auto-regression with exogenous variables (PARX) model for predicting daily energy consumption. This prediction is then compared to the actual consumption using an L1 distance. The distribution of these L1 distances is modeled using a Gaussian distribution (specifically, a log-normal distribution of the distances themselves). An anomaly is flagged if the probability of observing the current L1 distance falls below a user-defined threshold. The PARX model incorporates exogenous variables like temperature, which are known to influence energy consumption.
  • Lambda Architecture Implementation:
    • Speed Layer: Utilizes Spark Streaming for real-time processing of incoming smart meter data. It ingests data streams, performs anomaly detection using the latest models, and outputs detected anomalies.
  • Batch Layer: Employs Spark for batch processing. This layer is responsible for training and updating the anomaly detection models using historical data. It reprocesses the entire dataset periodically to ensure models are up-to-date.
  • Serving Layer: Uses PostgreSQL to store the trained detection models and the detected anomalies. This layer serves the models to the speed layer for real-time detection and can also be used for customer notifications.

Datasets

  • Training and Evaluation: A real-world residential electricity consumption dataset comprising 27,300 time series, with a two-year length and hourly resolution, was used for evaluation.
  • Scalability Testing: A synthetic dataset was generated, seeded by the real-world data, to evaluate scalability. This synthetic dataset was scaled up to one terabyte, representing over twenty million time series.

Key Insights and Ideas

  • Lambda Architecture for Anomaly Detection: The paper demonstrates that the Lambda architecture is a suitable paradigm for anomaly detection systems that require both real-time detection and efficient model refreshment. It effectively handles the trade-off between latency and throughput.
  • PARX with Exogenous Variables: Incorporating exogenous variables like temperature into the PARX model significantly improves the accuracy of energy consumption prediction, which is crucial for effective anomaly detection.
  • Log-Normal Distribution of L1 Distances: The observation that the L1 distances between predicted and actual consumption follow a log-normal distribution is a key insight that informs the choice of statistical modeling for anomaly detection.
  • Model Update Frequency: Frequent model updates (e.g., daily) are shown to improve detection accuracy by reducing the number of detected anomalies, likely due to better adaptation to changing consumption patterns.
  • Scalability of the System: The proposed system exhibits strong scalability, with detection and training processes scaling near linearly with the volume of data and the number of executors.

Key Results

  • Anomaly Detection Accuracy: The proposed PARX prediction and statistical method resulted in a notable decrease in the number of reported anomalies compared to a standard boxplot approach, suggesting a higher accuracy in identifying true anomalies.
  • Scalability:
    • Scale-Out Experiment: Execution time for both batch model training and real-time anomaly detection decreased as the number of executors increased, up to a point where overhead became a factor. For batch training, 64 executors were near optimal, while for real-time detection, 128 executors showed benefits.
  • Increased Data Load Experiment: Detection and training processes scaled near linearly with the quantity of time series. For instance, processing eight million time series (275 GB) took less than two minutes.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the boxplot approach, while simple, can lead to many false positives.
  • The current system's anomaly detection is primarily based on deviations from expected patterns, and identifying the reasons for anomalies is not explicitly addressed.
  • Future Work:
    • Implement a scheduling system to coordinate batch and real-time jobs within the same cluster.
  • Explore methods to detect a wider range of anomalies, including missing values, negative energy consumption, and device errors.
  • Extend the system to support additional data types such as gas, heating, and water consumption.
  • Develop corresponding detection algorithms for these new data types.

Summary updated

BibTeX
@inproceedings{liu2016online,
  author = {Xiufeng Liu and Xiufeng Liu and Nadeem Iftikhar and Per Sieverts Nielsen and Alfred Heller},
  title = {{Online Anomaly Energy Consumption Detection Using Lambda Architecture}},
  booktitle = {Dawak},
  year = {2016},
  doi = {10.1007/978-3-319-43946-4\_13},
}
Figure