Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation

Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, Dong Wang

Introduction

Non-Intrusive Load Monitoring (NILM), or energy disaggregation, is a critical technology for modern smart grids. By analyzing a single aggregate power signal from a main utility meter, NILM algorithms can estimate the energy consumption of individual appliances. This provides consumers with detailed usage insights and helps utility providers manage demand more effectively.

While deep learning architectures—particularly Transformers—have significantly advanced the state-of-the-art in NILM, they face two primary challenges. First, the standard self-attention mechanism possesses a quadratic computational complexity ($O(l^2)$) relative to sequence length, which limits its deployment on resource-constrained smart sensors. Second, Transformers often lack the "local inductive bias" necessary to capture the short-term, fine-grained signatures of specific appliances, such as the distinct power spike of a kettle or the cycling of a refrigerator compressor.

To address these issues, the ELTransformer (Efficient Localness Transformer) introduces a novel architecture that combines linear-complexity global attention with specialized local attention heads, offering a high-performance yet lightweight solution for energy disaggregation.

Methodology

The ELTransformer architecture improves upon the standard Transformer block through three primary innovations designed to balance global context with local precision.

1. Linear Attention Mechanism

To overcome the $O(l^2)$ bottleneck, ELTransformer utilizes a linear attention mechanism. By altering the order of matrix multiplication—applying normalization functions to the Query ($Q$) and Key ($K$) matrices before computing the product—the model shifts the complexity to $O(l)$. This allows the model to process long power-signal sequences with significantly lower memory and computational overhead, making it more suitable for edge devices.

2. Localness Modeling

Unlike standard Transformers that treat all data points with equal potential connectivity, ELTransformer employs a multi-head approach that incorporates "sparse local attention heads." While some heads focus on the global context of the entire sequence, these local heads use a sliding window ($l_{win}$) to focus exclusively on neighboring data points. This dual-pathway approach ensures the model captures both the broad patterns of household energy use and the specific "fingerprints" of individual devices.

3. Relative Position Encodings

The model adopts a Sequence-to-Point (Seq2Point) framework, where it predicts the power consumption of an appliance at a single midpoint based on a surrounding window of aggregate data. To support this, the authors implemented symmetric relative positional encodings, which help the model understand the temporal distance of signal events relative to the target prediction point.

Experimental Evaluation

The ELTransformer was evaluated using two benchmark datasets: UK-DALE (UK Domestic Appliance-Level Electricity) and REDD (Reference Energy Disaggregation Data). The study focused on five common household appliances: dishwasher, fridge, kettle, microwave, and washing machine. To test the model's robustness, the researchers conducted "unseen house" testing, where the model was trained on one set of homes and evaluated on a completely different residence.

Key Results and Insights

Superior Accuracy and Efficiency

On the UK-DALE dataset, the ELTransformer achieved an 8.2% improvement in Mean Absolute Error (MAE) compared to previous state-of-the-art models. Remarkably, this performance gain did not come at the cost of speed. Compared to the BERT4NILM baseline, the ELTransformer was 32.3% faster during inference and 27.8% faster during training.

Lightweight Architecture

Parameter efficiency is a standout feature of the ELTransformer. With only 1.91M parameters, it is significantly smaller than traditional CNN-based models like Seq2Point (30.7M) or SGNet (61.4M). Furthermore, as input sequence lengths increase, the ELTransformer’s inference time remains nearly flat, whereas standard Transformers experience an exponential spike in processing time.

The Importance of Inductive Bias

The research highlights that "pure" Transformers may be too flexible for the specific task of NILM. By introducing the constraint of local windows, the model more effectively identifies status changes in complex, multi-state appliances like washing machines. The synergy between global and local attention proves to be the "sweet spot" for signal disaggregation.

Limitations and Future Work

Despite its success, the ELTransformer faces some challenges. The optimal local window size ($l_{win}$) appears to vary by appliance; for instance, a kettle's signature is much shorter than a refrigerator's cycle, suggesting that a multi-scale window approach might be beneficial. Additionally, while the model leads in average performance, some simpler CNN baselines still occasionally outperform it on specific metrics for complex appliances like the washing machine.

Future research will focus on deploying the ELTransformer on actual low-power smart sensor hardware to validate its performance in real-time, "in-the-wild" scenarios.

Summary updated

Abstract

Modern smart sensor-based energy management systems leverage non-intrusive load monitoring (NILM) to predict and optimize appliance load distribution in real-time. NILM, or energy disaggregation, refers to the decomposition of electricity usage conditioned on the aggregated power signals (i.e., smart sensor on the main channel). Based on real-time appliance power prediction using sensory technology, energy disaggregation has great potential to increase electricity efficiency and reduce energy expenditure. With the introduction of transformer models, NILM has achieved significant improvements in predicting device power readings. Nevertheless, transformers are less efficient due to O(l^2) complexity w.r.t. sequence length l. Moreover, transformers can fail to capture local signal patterns in sequence-to-point settings due to the lack of inductive bias in local context. In this work, we propose an efficient localness transformer for non-intrusive load monitoring (ELTransformer). Specifically, we leverage normalization functions and switch the order of matrix multiplication to approximate self-attention and reduce computational complexity. Additionally, we introduce localness modeling with sparse local attention heads and relative position encodings to enhance the model capacity in extracting short-term local patterns. To the best of our knowledge, ELTransformer is the first NILM model that addresses computational complexity and localness modeling in NILM. With extensive experiments and quantitative analyses, we demonstrate the efficiency and effectiveness of the the proposed ELTransformer with considerable improvements compared to state-of-the-art baselines.
BibTeX
@inproceedings{yue2022efficient,
  author = {Zhenrui Yue and Huimin Zeng and Ziyi Kou and Lanyu Shang and Dong Wang},
  title = {{Efficient Localness Transformer for Smart Sensor-Based Energy Disaggregation}},
  booktitle = {DCOSS},
  year = {2022},
  doi = {10.1109/dcoss54816.2022.00035},
}
Figure