Abstract
In this paper, we introduce a nonparametric end-to-end method for probabilistic forecasting of distributed renewable generation outputs while including missing data imputation. Firstly, we employ a nonparametric probabilistic forecast model utilizing the long short-term memory (LSTM) network to model the probability distributions of distributed renewable generations' outputs. Secondly, we design an end-to-end training process that includes missing data imputation through iterative imputation and iterative loss-based training procedures. This two-step modeling approach effectively combines the strengths of the nonparametric method with the end-to-end approach. Consequently, our approach demonstrates exceptional capabilities in probabilistic forecasting for the outputs of distributed renewable generations while effectively handling missing values. Simulation results confirm the superior performance of our approach compared to existing alternatives.
Introduction
The transition to green energy relies heavily on Distributed Generation (DG), such as rooftop solar and small-scale wind farms. However, managing a power grid with these sources is challenging due to their stochastic nature. Effective grid operation requires probabilistic forecasts—ranges of likely outcomes rather than single-point estimates. A significant hurdle in generating these forecasts is the prevalence of missing sensor data caused by equipment failures or communication issues.
Historically, researchers have used a "two-phase" approach: first imputing missing values using simple algorithms (like linear interpolation or K-Nearest Neighbors) and then feeding the completed dataset into a forecasting model. This separation often leads to error propagation, where inaccuracies in the imputation phase compromise the final forecast's reliability.
Methodology
To address these issues, this research proposes a Nonparametric End-to-End framework. The core of the method is a Long Short-Term Memory (LSTM) network designed for quantile regression. Unlike parametric models that assume a specific data distribution (e.g., Gaussian), this nonparametric approach directly predicts specific quantiles (e.g., 10th, 50th, 90th percentiles), making it highly flexible for the irregular distributions typical of wind and solar power.
Iterative Imputation and Training
The model integrates imputation directly into the training loop. When a data point is missing, the model uses its own 0.5-quantile (median) forecast from the previous time step as the input for the current step. This "self-correction" ensures the forecasting and imputation tasks are optimized toward a unified goal.
The training utilizes a pinball loss function. Crucially, the loss is only calculated at time spots where actual observations are available, preventing the model from being penalized for its own imputed values while allowing it to learn the underlying temporal patterns.
Experimental Setup
The model was validated using real-world data from the Bluff Range Wind Farm in South Australia (52.5 MW capacity). The dataset spanned 2018 to 2020 with 5-minute resolution. To test robustness, a 25% missing data rate was artificially introduced using a "Missing Completely At Random" (MCAR) mechanism.
Results and Discussion
The proposed method was compared against traditional two-phase benchmarks (Linear Interpolation-LSTM and KNN-LSTM) and a parametric end-to-end model (Auto-LSTM).
As shown in the results, the proposed method significantly outperforms the alternatives:
- Reliability ($R_l$): Achieved a deviation of 3.42%, compared to 11.4% for LI-LSTM.
- Sharpness ($S_l$): Produced the narrowest prediction intervals (0.0415), indicating higher certainty.
- Skill Score ($Sk_l$): Achieved the best overall balance of reliability and sharpness with a score of -0.276.
The nonparametric nature of the model proved essential, as parametric models like Auto-LSTM often struggle with the "spiky" and non-Gaussian noise inherent in wind power data.
Conclusion
By treating imputation as an intrinsic part of the training process rather than a preprocessing step, this end-to-end approach effectively mitigates error propagation. Future research will focus on handling "block" missingness (extended sensor outages) and optimizing the computational efficiency of the iterative training process.
Summary updated