Defining 'Good': Evaluation Framework for Synthetic Smart Meter Data

Sheng Chai, Gus Chadney, Charlot Avery, Phil Grunewald, Pascal Van Hentenryck, Priya L. DontiarXiv 2024 · DOI · arXiv

Introduction

To achieve "Net Zero" targets, grid operators require granular, high-frequency electricity demand data to manage a grid increasingly dependent on volatile renewable energy sources. However, smart meter data—typically collected at 30-minute intervals—presents a significant privacy risk. A household's energy "fingerprint" can inadvertently reveal occupancy patterns, appliance usage, and socioeconomic status.

Synthetic data, generated by machine learning models to mimic real patterns without exposing individual records, offers a potential solution. However, defining what constitutes "good" synthetic data in the energy sector remains a challenge. Existing frameworks from other domains often overlook the unique temporal dependencies and "spikiness" of power consumption. This paper proposes a rigorous three-pillar evaluation framework—Fidelity, Utility, and Privacy—specifically designed for smart meter data.

The Three-Pillar Framework

1. Fidelity: Statistical Realism

Fidelity measures how closely the synthetic data distribution matches the real data. The authors employ metrics such as the Autocorrelation Function (ACF) to ensure temporal dependencies are maintained and Maximum Mean Discrepancy (MMD) to compare the distributions of consumption peaks and clusters.

Quantile plots comparing real and synthetic data distributions across various percentiles.

As illustrated in the quantile plots above, the generative model (Faraday) successfully captures the general trend of the data across the mean and median (50th percentile). However, maintaining fidelity at the extremes (1st and 99th percentiles) remains a significant challenge, particularly when privacy constraints are introduced, as the model struggles to replicate rare "spiky" events without memorizing them.

2. Utility: Practical Application

Utility evaluates whether the synthetic data is fit for purpose. The authors use a "Train on Synthetic, Test on Real" (TSTR) methodology. If a forecasting or classification model trained solely on synthetic data performs accurately when applied to real-world holdout data, the synthetic dataset is considered high-utility. In seasonal classification tasks, the authors found that synthetic data could achieve near-real-world accuracy levels.

3. Privacy: Beyond Standard Metrics

Privacy is the most critical pillar. The authors implement Differential Privacy (DP) using the DP-SGD algorithm, which injects controlled noise during the training process to prevent the model from over-focusing on any single data point.

The Novel "Outlier-Poisoning" Crash Test

The paper’s primary contribution is the identification that standard Membership Inference Attacks (MIA) are often too weak for energy data. Because most household profiles are statistically similar, an attacker cannot easily distinguish if a specific "average" house was in the training set, leading to a false sense of security.

Geometric visualization of the outlier-poisoning attack and the membership threshold.

To address this, the authors propose "Outlier-Poisoned Attacks." By injecting "implausible" outliers—such as a household consuming 20 times the average—into the training set, they create a "crash test" for the model. As shown in the visualization above, if these unique, poisoned records fall within the reconstructed threshold of the synthetic output, the model has failed to provide adequate privacy. This method reveals vulnerabilities that standard tests often miss.

Key Insights and Results

  • The Epsilon (ε) Trade-off: In Differential Privacy, ε represents the privacy loss. The authors found that while ε=1.0 provides robust protection, it significantly degrades the fidelity of the data. Conversely, an ε=8.0, often considered acceptable in general machine learning, provided almost no protection against reconstruction attacks in this context.
  • Privacy vs. Data Volume: Increasing the size of the training set improves fidelity and utility but does not inherently improve privacy. Privacy remains a strict function of the noise added via DP, not the volume of data.
  • Hierarchical Fidelity: The authors emphasize that synthetic data must look realistic not just at the household level, but also when aggregated to the substation level to respect the laws of physics and grid operations.

Conclusion and Future Work

While the proposed framework provides a robust baseline, limitations remain. The current study focuses on the Low Carbon London (LCL) dataset, which lacks demographic information. Future research must investigate "Attribute Inference"—whether synthetic data accidentally leaks hidden traits like income or age. Furthermore, the authors advocate for moving toward "Predict-then-Optimize" utility metrics, evaluating synthetic data based on its performance in grid optimization tasks like battery placement or EV charging management.

Summary updated

BibTeX
@misc{chai2024defining,
  author = {Sheng Chai and Gus Chadney and Charlot Avery and Phil Grunewald and Pascal Van Hentenryck and Priya L. Donti},
  title = {{Defining 'Good': Evaluation Framework for Synthetic Smart Meter Data}},
  year = {2024},
  eprint = {2407.11785},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.11785},
}
Figure