Household Electricity Consumption Data Cleansing

Guoming Tang, Kui Wu, Jian Pei, Jiuyang Tang, Jingsheng Lei

Abstract

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present a new appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as B-spline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also demonstrate that our method is robust in various tests. Our method provides a highly feasible and reliable solution to an emerging industry application.

1. Introduction

Modern smart grids rely heavily on "load curve data"—periodic recordings of electricity consumption—for billing, grid operations, and energy generation planning. However, this data is frequently corrupted by transmission errors, malfunctioning meters, and wireless packet loss. Traditional "appliance-oblivious" methods, such as B-spline smoothing or regression, primarily target statistical outliers. These methods often fail to detect "hidden" corrupted data: values that appear statistically normal but are physically impossible given the specific appliances in a household.

Figure 1: Conceptual illustration of hidden corrupted data. When the aggregated power falls outside the valid ranges defined by appliance states, it is identified as corrupted.

2. Methodology

The proposed appliance-driven approach formulates the Corrupted Data Identification Problem (CDIP) by incorporating known power ranges (minimum and maximum wattage) of a building's appliances.

2.1 The Virtual Appliance

To detect errors, a mathematical "virtual appliance" is introduced. If the sum of real appliance power consumption cannot match the recorded load within physical constraints, the virtual appliance "activates" to bridge the gap. By minimizing the $l_1$-norm of this virtual appliance's power, the model identifies the sparsest and most likely instances of data corruption.

2.2 Sequential Local Optimization Algorithm (SLOA)

Since solving the CDIP globally is NP-complete, the authors developed the Sequential Local Optimization Algorithm (SLOA). SLOA partitions the problem into small, overlapping time windows. An Overlap Index ($O$) heuristic is used to determine optimal window sizes based on appliance power range overlaps, ensuring computational efficiency without compromising accuracy.

3. Experimental Evaluation

The methodology was validated using two distinct datasets:

  1. Real-World Trace Data: Collected over two months from a laboratory at the University of Victoria (UVic), featuring 13 appliances sampled every 10 seconds via ZigBee sensors.
  2. Synthetic Data: Generated through Monte Carlo simulations to test the algorithm's scalability with 50 appliances and various simulated data corruption patterns.

4. Results and Discussion

The SLOA method demonstrates significant improvements over traditional statistical smoothing by prioritizing physical ground truths over statistical patterns.

Figure 6: Performance of the SLOA algorithm. The top plot shows the load curve with estimated physical bounds, identifying corrupted spikes. The bottom plot quantifies the "Corrupted Degree" over time.
  • Precision: SLOA achieved a precision of up to 95.83% on real-world data, vastly outperforming the ~50% precision of B-spline smoothing by minimizing false alarms.
  • Consecutive Errors: Unlike traditional methods that attempt to "fit" long streaks of corrupted data (e.g., a meter stuck at zero), SLOA correctly identifies these as errors.
  • Robustness: The algorithm is resilient to "cold starts," successfully recovering and identifying errors even without initial knowledge of appliance states.

5. Conclusion

This appliance-driven approach shifts the focus from statistical patterns to physical constraints. While privacy concerns regarding appliance-level data sharing remain a hurdle, future work will explore privacy-preserving versions of this cleansing method. By leveraging domain knowledge of the demand side, SLOA provides a reliable solution for maintaining high-quality load curve data in modern power systems.

Summary updated

Abstract

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present a new appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as B-spline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also demonstrate that our method is robust in various tests. Our method provides a highly feasible and reliable solution to an emerging industry application.
BibTeX
@misc{tang2013household,
  author = {Guoming Tang and Kui Wu and Jian Pei and Jiuyang Tang and Jingsheng Lei},
  title = {{Household Electricity Consumption Data Cleansing}},
  year = {2013},
  eprint = {1307.7757},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1307.7757},
}
Figure