TL;DR
This paper introduces the Temporal Structure-Preserving Transformer (TSPT), a novel deep learning architecture designed to improve the accuracy of industrial load forecasting. By decomposing complex multi-target load series and effectively integrating diverse exogenous data, TSPT offers a more robust and scalable solution for energy management in industrial settings.
What Problem Does This Paper Address?
Accurate forecasting of electricity consumption in industrial parks is crucial for efficient energy management and operational planning. However, industrial load data presents unique challenges: it's often multi-target (multiple buildings or areas), influenced by a wide array of dynamic exogenous factors (weather, production schedules, efficiency measures), and exhibits complex temporal dependencies across various scales. Existing forecasting models, including traditional time series methods and even advanced deep learning architectures like standard Transformers, often struggle to effectively handle this complexity. They may fail to decompose multi-target series, integrate diverse external data seamlessly, or capture long-range temporal patterns without significant computational cost. This gap hinders optimal energy management and decision-making in industrial environments.
Methods
The TSPT architecture tackles these challenges through several key innovations:
- Temporal Structure-Preserving Transformer (TSPT): The core of the approach. It decomposes multi-target load series into univariate series for parallel processing.
- Instance Normalization (RevIN): Applied at the beginning to stabilize training by normalizing input scales and handling potential non-stationarity, with a reversible mechanism to recover original scales for predictions.
- Feature Extraction Perceptrons (TSFP & EFP): Parallel Multi-Layer Perceptrons (MLPs) that map raw target load data and exogenous factors into a shared, lower-dimensional latent space, facilitating multimodal fusion.
- Temporal Patching: Divides the latent representations into smaller, overlapping patches to reduce sequence length, improving computational efficiency and enabling localized temporal pattern learning. This reduces the quadratic complexity of standard self-attention.
- Gated Feature Fusion (GFF): A novel mechanism that adaptively combines patched target and exogenous features. A learnable gating tensor modulates the fusion, allowing the model to selectively emphasize or suppress exogenous influences based on context, thus capturing complex cross-modal interactions.
- Multilayer Temporal Attention Encoder: A stack of standard Transformer encoder layers that process the fused patch representations to capture intricate, long-range temporal dependencies.
Datasets
The study utilizes a 6-year (2016-2021) power load dataset from four building types (office, residential, commercial, public) in an industrial park in Suzhou, China. This dataset includes hourly power load observations and hourly exogenous factor observations (temperature and humidity). The data was preprocessed to handle missing values using the Multiple Imputation by Chained Equations (MICE) method. The dataset was split into 80% for training and 20% for testing.
Key Insights and Ideas
- Decomposition is Key: Breaking down complex multi-target industrial load series into simpler univariate components is crucial for efficient and effective modeling.
- Adaptive Fusion Matters: Simply including exogenous data isn't enough; an adaptive mechanism like GFF is needed to intelligently integrate it, allowing the model to learn when and how much to rely on external factors.
- Patching for Efficiency: Temporal patching is a smart way to make Transformer models computationally feasible for long industrial time series without sacrificing the ability to capture local patterns.
- Instance Normalization is Critical: For industrial data, which can have varying scales and non-stationarity, RevIN is not just helpful but essential for stable training and accurate predictions.
- No One-Size-Fits-All: The performance of linear vs. non-linear models varies significantly across different building types, highlighting the need for models that can handle both linear and non-linear relationships simultaneously.
Key Results
- TSPT consistently outperformed 19 baseline methods (including LSTM, GRU, DLinear, Informer, Autoformer, and PatchTST) across various metrics (MSE, MAE, PCC) for both 24-h and 48-h forecast horizons.
- With exogenous factors, TSPT achieved the best MSE (7.568 × 10⁻³), MAE (52.096), and PCC (84.215%) for a 24-h horizon. For a 48-h horizon, it achieved the lowest MSE (10.193 × 10⁻³), MAE (61.676), and highest PCC (80.501%).
- The inclusion of exogenous factors generally improved TSPT's performance, demonstrating its ability to leverage external information.
- In cross-correlation analyses, TSPT showed remarkable robustness, maintaining leading performance across different building types and variations in data characteristics.
- Ablation studies revealed that Instance Normalization was the most critical component, followed by the Temporal Patching and the Transformer Encoder, highlighting the synergistic contribution of each module.
Limitations and Future Work
The authors acknowledge that the empirical validation was primarily based on a single industrial park dataset. Future work should involve evaluating TSPT on more diverse datasets from different geographical locations and industrial sectors to assess its generalizability. They also suggest exploring alternative fusion strategies (e.g., cross-attention) and investigating the interpretability of the model, particularly the learned attention weights and gating tensor, to gain deeper insights into its decision-making process.
Summary updated