Glassoformer: A Query-Sparse Transformer for Post-Fault Power Grid Voltage Prediction

Yunling Zheng, Carson Hu, Guang Lin, Meng Yue, Bao Wang, Jack Xin

TL;DR

This paper introduces GLassoformer, a novel transformer architecture that significantly improves computational efficiency for predicting post-fault power grid voltage. By employing group Lasso regularization to create sparse queries in the self-attention mechanism, GLassoformer achieves superior accuracy and stability compared to existing methods, making it a promising tool for real-time grid management.

What Problem Does This Paper Address?

Power grids are critical infrastructure, and predicting their behavior after faults (like line failures or natural disasters) is crucial for maintaining stability and preventing cascading blackouts. Traditional methods involve complex simulations that are too slow for real-time decision-making. While machine learning, particularly transformers, has shown promise in learning from sensor data, standard transformers suffer from quadratic computational costs with respect to sequence length, making them impractical for long-term predictions. Furthermore, redundant queries in transformers can even degrade prediction accuracy. This paper aims to bridge this gap by developing a more efficient and accurate transformer model for post-fault voltage prediction.

Methods

The core innovation of GLassoformer lies in its modified self-attention mechanism. Standard transformers use dense queries, keys, and values. GLassoformer introduces group Lasso regularization to the query weight matrix. This regularization encourages sparsity, effectively selecting only a small subset of "important" queries. This "query sparsity" dramatically reduces the computational complexity of the attention mechanism, particularly the matrix multiplications involved in computing the attention matrix and the final output.

The architecture itself is an encoder-decoder transformer. The encoder processes input signals (voltage and current from buses and lines) and temporal information, while the decoder uses this encoded information along with past observations to predict future voltage trajectories. The model also incorporates 1D convolutional layers within the encoder and decoder to capture local temporal features, and uses an ELU activation function for smoother gradients. The training process leverages a Relaxed Group-wise Splitting Method (RGSM), which is designed to handle the group Lasso penalty effectively and ensure convergence.

Datasets

The paper evaluates GLassoformer on a dataset derived from the York/New England 16-generator 68-bus power system. This dataset contains voltage and current measurements from buses and lines over 10-second intervals following fault events. The data is split into training, validation, and testing sets with a ratio of 1000/350/750 fault events, respectively.

Key Insights and Ideas

  • Query Sparsity is Key: The paper demonstrates that not all queries in a transformer's self-attention are equally important. By strategically sparsifying queries using group Lasso, significant computational gains can be achieved without sacrificing accuracy.
  • Efficiency Without Compromise: GLassoformer achieves a remarkable balance between computational efficiency and prediction accuracy. This is crucial for real-time applications where speed is paramount.
  • Structured Sparsity Matters: The use of group Lasso, as opposed to unstructured Lasso, encourages sparsity at a group level, which is more aligned with the structured nature of the query matrix in transformers.
  • Transformer Limitations Addressed: The paper directly tackles the quadratic complexity and potential redundancy issues inherent in standard transformers, offering a practical solution for long sequence modeling.

Key Results

GLassoformer significantly outperforms existing benchmark algorithms, including Informer, Lasso2, and Prony's method, in terms of accuracy (lower Mean Squared Error (MSE) and Mean Absolute Error (MAE)) for post-fault voltage prediction. For instance, in one experiment, GLassoformer achieved an MSE of 3.189 x 10⁻⁵, compared to Informer's 3.662 x 10⁻⁵ and Lasso2's 3.532 x 10⁻⁵.

In terms of efficiency, GLassoformer is faster at inference than Informer. While training times are comparable on this specific dataset, the model's parameter size is smaller than Informer's. The paper also shows that GLassoformer's predictions are more stable, exhibiting fewer spurious spikes compared to Informer.

Limitations and Future Work

The authors acknowledge that while GLasso computation without query vector sampling doesn't reduce the O(N²) complexity of attention computation during training, it does so at inference. For their moderate-sized dataset and network, training times were similar with or without random sampling. However, they highlight that GLassoformer is faster and more accurate than Informer during testing.

Future work could explore applying GLassoformer to larger and more complex power grid systems, as well as investigating its performance on other time-series prediction tasks where long-range dependencies and computational efficiency are critical. Further research into optimizing the group Lasso penalty and the RGSM training algorithm could also yield additional improvements.

Summary updated

Abstract

We propose GLassoformer, a novel and efficient transformer architecture leveraging group Lasso regularization to reduce the number of queries of the standard self-attention mechanism. Due to the sparsified queries, GLassoformer is more computationally efficient than the standard transformers. On the power grid post-fault voltage prediction task, GLasso-former shows remarkably better prediction than many existing benchmark algorithms in terms of accuracy and stability.
BibTeX
@inproceedings{zheng2022glassoformer,
  author = {Yunling Zheng and Carson Hu and Guang Lin and Meng Yue and Bao Wang and Jack Xin},
  title = {{Glassoformer: A Query-Sparse Transformer for Post-Fault Power Grid Voltage Prediction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=KUBnZngJMj},
}
Figure