Two-level clustering methodology for smart metering data

Leticia Arco, Gladys María Casas Cardoso, Ann NowéCuadernos 2020 · DOI

Abstract

Energy efficiency and sustainability are important factors to address in the context of smart cities. In this sense, a necessary functionality is to reveal various preferences, behaviors, and characteristics of individual consumers, considering the energy consumption information from smart meters. In this paper, we introduce a general methodology and a specific two-level clustering approach that can be used to group, considering global and local features, energy consumptions and productions of households. Thus, characteristic load and production profiles can be determined for each consumer and prosumer, respectively. The obtained results will be generally applicable and will be useful in a general business analytics context.

Introduction

The transition toward smart cities has transformed electricity grids from passive delivery systems into active, data-driven networks. Central to this evolution is the smart meter, which generates a "data deluge" of high-resolution readings (e.g., every 15 minutes). However, extracting actionable business intelligence from this raw data remains a significant challenge. Utilities need to understand diverse consumer behaviors to optimize grid stability and design effective demand-response programs. The core challenge is that energy behavior is multi-faceted: clustering based only on shape might group a small apartment with a commercial building, while clustering only by volume misses critical timing of peaks.

Methodology

The authors propose a comprehensive five-stage methodology to process and analyze smart meter data, moving from raw collection to refined consumer profiles.

The five-stage methodology for smart meter data processing.

The core of this framework is a two-level feature-based clustering approach:

  1. Level 1: Global Feature-Based Representation: This stage groups consumers by general behavior and consumption magnitude using summary statistics (mean, min, max, total sum). The k-means algorithm is employed for its efficiency in handling large datasets.
  2. Level 2: Local Feature-Based Representation: This stage refines the initial clusters by examining temporal patterns or "peaks." Using windowed statistics (e.g., hourly summaries for daily profiles), Complete Linkage hierarchical clustering identifies specific behavioral prototypes.

By focusing on statistical features rather than raw time-series data, the method achieves significant dimensionality reduction and inherent noise filtering.

Key Insights and Ideas

A primary finding of this research is the critical distinction between Magnitude and Shape. Traditional single-level clustering often fails to distinguish between different types of users if their peak usage times align.

Comparison of time series similarity based on shape versus change.

As illustrated in the figure above, time series can be similar in shape but different in magnitude (top panel), or similar in their pattern of change despite different baseline levels (bottom panel). The two-level approach ensures that both "how much" and "when" are captured.

Other key insights include:

  • The Weekend Effect: Consumer behavior varies so significantly between weekdays and weekends that data must be stratified prior to clustering to maintain accuracy.
  • Prosumer Seasonality: For households with solar production, the methodology successfully categorized "high" vs. "low" producers within the expected seasonal trends (peaking between March and September).

Results and Discussion

The study applied this methodology to a real-world dataset from Eandis (Belgium), covering 2,928 homes over a full year. The system successfully identified several representative prototypes:

  • Daily Prototypes: Consumers were categorized into High, Medium, and Low consumption groups, with sub-clusters identifying specific morning or afternoon peak behaviors.
  • Yearly Prototypes: The researchers grouped 375 prosumers by annual output while preserving shared seasonal production characteristics.
  • Computational Efficiency: Reducing 96 daily data points into a compact set of features allowed for the use of sophisticated clustering algorithms that are typically too slow for large-scale grid data.

Conclusion

While the current model uses Euclidean distance and a specific set of eight statistical features, future work could explore metrics like Dynamic Time Warping (DTW) or more complex features (e.g., entropy, seasonality indices). Adapting this offline analysis for real-time stream clustering remains a vital next step for detecting grid anomalies as they occur.

Summary updated

Abstract

Energy efficiency and sustainability are important factors to address in the context of smart cities. In this sense, a necessary functionality is to reveal various preferences, behaviors, and characteristics of individual consumers, considering the energy consumption information from smart meters. In this paper, we introduce a general methodology and a specific two-level clustering approach that can be used to group, considering global and local features, energy consumptions and productions of households. Thus, characteristic load and production profiles can be determined for each consumer and prosumer, respectively. The obtained results will be generally applicable and will be useful in a general business analytics context.
BibTeX
@inproceedings{arco2020two,
  author = {Leticia Arco and Gladys María Casas Cardoso and Ann Nowé},
  title = {{Two-level clustering methodology for smart metering data}},
  booktitle = {Cuadernos},
  year = {2020},
  doi = {10.11144/javeriana.cao33.tlcms},
}
Figure