Evaluation of Scikit-Learn Machine Learning Algorithms for Improving CMA-WSP v2.0 Solar Radiation Prediction

Dan Wang, Yanbo Shen, Dong Ye, Yanchao Yang, Xuanfang Da, Jingyue MoAtmosphere 2024 · DOI

TL;DR

This paper evaluates the performance of China's CMA-WSP v2.0 solar radiation forecasting system and demonstrates how machine learning models from scikit-learn can significantly improve its accuracy, especially under challenging overcast and rainy conditions. The study highlights that while CMA-WSP v2.0 shows diurnal and weather-dependent errors, models like K-Nearest Neighbors and Random Forests can effectively reduce these errors, making solar energy predictions more reliable.

What Problem Does This Paper Address?

Accurate solar radiation forecasting is crucial for integrating renewable energy sources like solar power into the grid. However, solar energy's inherent intermittency and variability pose significant challenges. While numerical weather prediction (NWP) models like China's CMA-WSP v2.0 provide forecasts, they often exhibit inaccuracies, particularly during periods of complex weather (overcast, rainy) and at specific times of the day (early morning, late evening). These errors can hinder efficient grid management and optimal resource allocation. This paper addresses the need to evaluate the performance of the CMA-WSP v2.0 system and explore how readily available machine learning techniques can be applied to mitigate its forecasting errors, thereby improving the reliability of solar radiation predictions.

Methods

The study employs a suite of machine learning algorithms from the Python scikit-learn library to refine the solar radiation forecasts generated by the CMA-WSP v2.0. The selected models include:

  • Linear Regression (LR)
  • Decision Trees (DT)
  • K-Nearest Neighbors (KNN)
  • Random Forests Regression (RFR)
  • Adaptive Boosting (AdaBoost)
  • Gradient Boosting Regression (GBR)

These models were trained using historical observed solar radiation data and forecasted meteorological variables from CMA-WSP v2.0. The researchers also investigated the impact of feature engineering by incorporating time-series data related to diurnal solar radiation variations as an additional input. The performance of both the original CMA-WSP v2.0 and the ML-enhanced models was evaluated using metrics such as Mean Absolute Percentage Error (MAPE), Absolute Percentage Error (APE), Bias Error (BE), and Pearson's correlation coefficient (R).

Datasets

The study utilized the following datasets:

  • CMA-WSP v2.0 Forecast Data: This includes various meteorological variables (longwave and shortwave radiation, cloud cover, temperature, humidity, wind, etc.) forecasted by the CMA-WSP v2.0 system.
  • Observed Solar Radiation Data: 15-minute interval solar radiation observations were collected from a photovoltaic power station in Guizhou Province, China.
  • National Meteorological Station Data: Precipitation and cloud cover data from a station located 13 km from the photovoltaic power station were used for weather condition classification.

The datasets covered periods from January 2022 to December 2022 and from August 2023 to November 2023. The earlier period was used for training, and the later period for validation and testing.

Key Insights and Ideas

  • Diurnal and Weather-Dependent Errors: The CMA-WSP v2.0's forecasting accuracy is not uniform throughout the day or across different weather conditions. Errors are significantly higher in the early morning (07:00-08:00) and late evening (17:00-18:00) compared to midday. Overcast and rainy conditions lead to substantially higher prediction errors than sunny or cloudy days.
  • ML Models Offer Significant Improvement: Applying scikit-learn ML models can drastically reduce the MAPE of CMA-WSP v2.0 forecasts, particularly in challenging conditions.
  • KNN and RFR Shine: K-Nearest Neighbors (KNN) and Random Forests Regression (RFR) emerged as the top-performing ML models, consistently improving predictions across all weather types. KNN showed particular strength in rainy conditions.
  • Feature Engineering Matters: Including time-series data related to the diurnal pattern of solar radiation as an input feature can further enhance the performance of ML models.
  • Underestimation/Overestimation Trends: CMA-WSP v2.0 tends to underestimate solar radiation in sunny and cloudy conditions and overestimate it in overcast and rainy conditions. ML models generally follow similar trends but with reduced error magnitudes.

Key Results

  • CMA-WSP v2.0 MAPE: The MAPE for CMA-WSP v2.0 varies significantly, from around 74% during midday (11:00-13:00) to a high of 193%-242% during early morning and late evening hours. In terms of weather, MAPEs were 59% (sunny), 68% (cloudy), 114% (overcast), and a striking 403% (rainy).
  • ML Model Performance:
    • KNN and RFR: These models achieved the lowest MAPEs, with KNN outperforming others, especially in rainy conditions. For example, in sunny conditions, RFR and KNN achieved MAPEs of 35% and 37% respectively, compared to CMA-WSP v2.0's 59%.
  • GBR: Showed good performance, reducing MAPE in all conditions except rainy.
  • AdaBoost and LR: Had limited success in reducing MAPE, though they did improve correlation coefficients.
  • Correlation Coefficients: ML models generally improved the correlation coefficient (R) between observed and forecasted solar radiation, with values often exceeding 0.7 for RFR and KNN.
  • Impact of Schemes: Two additional schemes (Scheme 2: adding minute of the day; Scheme 3: time-based labels) showed modest improvements (1-3% reduction in MAPE) for some ML models, but did not fully resolve issues in early morning/late evening or rainy conditions.

Limitations and Future Work

  • Diurnal Performance Gaps: Despite improvements, the ML models still struggled with the extreme errors of CMA-WSP v2.0 during early morning (07:00-08:00) and late evening (17:00-18:00) hours, and particularly in rainy conditions.
  • Economic Considerations: The authors acknowledge that the practical application of ML models for solar radiation forecasting needs to consider economic factors like data collection, high-performance computing, maintenance, and model updates.
  • Future Work: The study suggests experimenting with more schemes to further improve ML model performance. They also plan to apply these findings to other photovoltaic power plants and explore the economic feasibility of ML-based forecasting in real-world scenarios. The comparison with the WRF model using daily total solar radiation indicates that CMA-WSP v2.0 is competitive and valuable for business applications.

Summary updated

Abstract

This article aims to evaluate the performance of solar radiation forecasts produced by CMA-WSP v2.0 (version 2 of the China Meteorological Administration Wind and Solar Energy Prediction System) and to explore the application of machine learning algorithms from the scikit-learn Python library to improve the solar radiation prediction made by the CMA-WSP v2.0. It is found that the performance of the solar radiation forecasting from the CMA-WSP v2.0 is closely related to the weather conditions, with notable diurnal fluctuations. The mean absolute percentage error (MAPE) produced by the CMA-WSP v2.0 is approximately 74% between 11:00 and 13:00. However, the MAPE ranges from 193% to 242% at 07:00–08:00 and 17:00–18:00, which is greater than that observed at other daytime periods. The MAPE is relatively low (high) for both sunny and cloudy (overcast and rainy) conditions, with a high probability of an absolute percentage error below 25% (above 100%). The forecasts tend to underestimate (overestimate) the observed solar radiation in sunny and cloudy (overcast and rainy) conditions. By applying machine learning models (such as linear regression, decision trees, K-nearest neighbors, random forests regression, adaptive boosting, and gradient boosting regression) to revise the solar radiation forecasts, the MAPE produced by the CMA-WSP v2.0 is significantly reduced. The reduction in the MAPE is closely connected to the weather conditions. The models of K-nearest neighbors, random forests regression, and decision trees can reduce the MAPE in all weather conditions. The K-nearest neighbor model exhibits the most optimal performance among these models, particularly in rainy conditions. The random forest regression model demonstrates the second-best performance compared to that of the K-nearest neighbor model. The gradient boosting regression model has been observed to reduce the MAPE of the CMA-WSP v2.0 in all weather conditions except rainy. In contrast, the adaptive boosting (linear regression) model exhibited a diminished capacity to improve the CMA-WSP v2.0 solar radiation prediction, with a slight reduction in MAPE observed only in sunny (sunny and cloudy) conditions. In addition, the input feature selection has a considerable influence on the performance of the machine learning model. The incorporation of the time series data associated with the diurnal variation of solar radiation as an input feature can further improve the model’s performance.
BibTeX
@inproceedings{wang2024evaluation,
  author = {Dan Wang and Yanbo Shen and Dong Ye and Yanchao Yang and Xuanfang Da and Jingyue Mo},
  title = {{Evaluation of Scikit-Learn Machine Learning Algorithms for Improving CMA-WSP v2.0 Solar Radiation Prediction}},
  booktitle = {Atmosphere},
  year = {2024},
  doi = {10.3390/atmos15080994},
}
Figure