BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles

Yi Hu, Kai Ye, Hyeonjin Kim, Ning Lü

TL;DR

BERT-PIN adapts the BERT architecture—renowned for its success in Natural Language Processing—to "inpaint" missing segments in power grid load profiles. By treating time-series segments as words in a sentence, the framework captures complex, long-range dependencies and offers multiple plausible restoration candidates instead of a single deterministic guess.

The Challenge of Missing Data in Power Systems

In power system analysis, load profiles—the records of electricity consumption over time—are essential for infrastructure planning, demand response, and billing. However, data gaps, or Missing Data Segments (MDSs), are frequent due to sensor failures or communication issues.

Existing inpainting methods, such as those based on Generative Adversarial Networks (GANs) or Recurrent Neural Networks (RNNs), often struggle with:

  1. Long-range dependencies: Capturing how consumption patterns in the morning correlate with evening peaks.
  2. Uncertainty: Most models provide a single deterministic output. In reality, multiple consumption scenarios may be plausible during a data gap, and analysts benefit from understanding this range of possibilities.

The BERT-PIN Framework

The authors introduce BERT-PIN (BERT-powered Profile Inpainting Network). The core innovation lies in translating electrical time-series data into a format compatible with Transformer architectures.

The BERT-PIN architecture, featuring data adaptation, the core BERT model, and top candidate selection methods.

Methodology

  • Data Adaptation: Load and temperature profiles are segmented into line segments. Each segment is treated as a "word," a daily profile as a "sentence," and a weekly profile as a "paragraph."
  • Classification vs. Regression: Instead of predicting continuous values, BERT-PIN maps power values into 200 discrete tokens. This allows the model to output a probability distribution for each missing point, facilitating uncertainty quantification.
  • Multi-modal Input: Recognizing that weather is a primary driver of electricity use, the model fuses load data with ambient temperature data from NOAA.

Embracing Uncertainty: Iterative Top-X Selection

A standout feature of BERT-PIN is its ability to generate multiple plausible restoration candidates. While standard models use an argmax function to pick the single most likely value, BERT-PIN introduces an iterative "fork point" method.

Iterative Top-X selection using forking points to generate multiple plausible data candidates from both directions.

If the model is uncertain (indicated by a "flat" probability distribution), it identifies these "forking points" and generates secondary versions of the data. This process ensures the restored profiles maintain physical realism and autocorrelation, avoiding jagged or impossible transitions.

Evaluation and Key Results

The study utilized a robust dataset of 15-minute interval smart meter data from 8,000 residential households in North Carolina (2018–2020), paired with hourly NOAA temperature data.

  • Superior Accuracy: BERT-PIN outperformed the previous state-of-the-art GAN-based method (Load-PIN) by approximately 10% in overall accuracy.
  • Robustness in Peak Masking: Restoring data during peak usage hours is notoriously difficult. BERT-PIN outperformed benchmarks by 5% to 27% across various error metrics (RMSE, MPE) in these scenarios.
  • Context Length: The BERT-PIN-Week model performed significantly better than the "Day" model by capturing weekly patterns (e.g., weekday vs. weekend behavior), though it required more training time.

Conclusion and Future Directions

BERT-PIN demonstrates that the self-attention mechanism in Transformers is exceptionally effective at "reading" the context of load profiles. By providing "Top-2" or "Top-3" candidates, the model offers an ensemble of possibilities critical for applications like demand response baseline estimation.

Future work includes fine-tuning BERT-PIN for downstream tasks such as load disaggregation (identifying individual appliance usage) and super-resolution (enhancing data frequency).

Summary updated

Abstract

Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for profile inpainting, we segment the load and temperature profiles into line segments, treating each segment as a word and the entire profile as a sentence. We incorporate a top candidates selection process in BERT-PIN, enabling it to produce a sequence of probability distributions, based on which users can generate multiple plausible imputed data sets, each reflecting different confidence levels. We develop and evaluate BERT-PIN using real-world dataset for two applications: multiple MDSs recovery and demand response baseline estimation. Simulation results show that BERT-PIN outperforms the existing methods in accuracy while is capable of restoring multiple MDSs within a longer window. BERT-PIN, served as a pre-trained model, can be fine-tuned for conducting many downstream tasks, such as classification and super resolution.
BibTeX
@misc{hu2023bert,
  author = {Yi Hu and Kai Ye and Hyeonjin Kim and Ning Lü},
  title = {{BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles}},
  year = {2023},
  eprint = {2310.17742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17742},
}
Figure