A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics, and Performance Results for Operational Solar Flare Forecasting Systems

K. D. Leka, Sung‐Hong Park, K. Kusano, Jesse Andries, G. Barnes, Suzy Bingham, D. Shaun Bloomfield, Aoife E. McCloskey, Véronique Delouille, D. A. Falconer, P. T. Gallagher, Manolis K. Georgoulis, Yûki Kubo, Kangjin Lee, Sang‐Woo Lee, V. V. Lobzin, JunChul Mun, Sophie A. Murray, Tarek A. M. Hamad Nageem, Rami Qahwaji, Michael Sharpe, R. A. Steenburgh, Graham Steward, Michael TerkildsenThe Astrophysical Journal Supplement Series 2019 · DOI

TL;DR

This paper presents the first direct, comprehensive comparison of operational solar flare forecasting systems, establishing a robust methodology and performance benchmarks. While many systems demonstrated skill above a "no-skill" baseline, no single method emerged as universally superior, with performance heavily dependent on the specific flare definition and evaluation metric used.

What Problem Does This Paper Address?

Solar flares are powerful bursts of energy from the Sun that can disrupt technology on Earth, making accurate forecasting crucial for space weather preparedness. Despite advances in data and algorithms, a standardized way to rigorously compare the performance of different operational forecasting systems has been lacking. This paper addresses this gap by developing a common evaluation framework and applying it to a suite of leading operational forecasting methods, aiming to establish benchmarks and understand the factors influencing their performance.

Methods

The authors established a standardized testing interval (January 2016 - December 2017) and common event definitions (C1.0+, M1.0+ flares with 24-hour validity and zero latency) to ensure fair comparison. They employed a range of quantitative metrics, including:

  • Probabilistic metrics: Reliability Plots (Attribute Diagrams) and Brier Skill Score (BSS) to assess how well predicted probabilities match observed frequencies.
  • Dichotomous metrics: ROC curves (Probability of Detection vs. Probability of False Detection), True Skill Statistic (TSS), Appleman Skill Score (ApSS), Equitable Threat Score (ETS), and Frequency Bias (FB) to evaluate performance based on thresholded forecasts.
  • "No-skill" reference forecasts: The authors utilized a 120-day prior climatology as a baseline "unskilled" forecast to contextualize the performance of operational systems.

Datasets

The study utilized data from the Solar Dynamics Observatory (SDO) mission, specifically the Helioseismic and Magnetic Imager (HMI) and the Geostationary Operational Environmental Satellite (GOES) X-Ray Sensor (XRS). The testing interval covered the period from January 1, 2016, to December 31, 2017. The paper also references historical flare data and sunspot catalogs used for training by the individual forecasting methods.

Key Insights and Ideas

  • No single "best" method: The most striking finding is the absence of a universally superior forecasting method. Performance is highly context-dependent, varying significantly with the chosen flare event definition (e.g., C-class vs. M-class flares) and the specific metric used for evaluation.
  • Importance of methodology: The paper emphasizes that a well-defined and consistent methodology is crucial for meaningful comparisons. This includes standardized event definitions, testing intervals, and a suite of appropriate metrics.
  • Operational constraints matter: The evaluation acknowledges the limitations of operational forecasting, such as the need for timely forecasts and the potential for data issues, which can influence performance.
  • Skill above "no-skill": A significant number of operational methods consistently demonstrated skill above the established "no-skill" baseline, indicating progress in the field.
  • Subtle differences in implementations: Even variations of the same core method (e.g., using different magnetogram data or including prior flare history) showed subtle but discernible differences in performance across various metrics.

Key Results

  • Broadly consistent performance: While specific rankings varied, most methods performed similarly to each other across multiple metrics, especially for M1.0+ flares.
  • Varied performance across metrics: Some methods excelled in certain metrics (e.g., TSS/PSS) while performing less well in others (e.g., ApSS), highlighting the need to consider a range of evaluation tools.
  • Underforecasting tendency: For larger flares (M1.0+), the majority of methods tended towards underforecasting (Frequency Bias < 1.0).
  • Impact of data sources: Methods utilizing vector magnetic field data showed slight improvements over those using line-of-sight magnetograms in some cases.
  • No method scored above 0.5 on all metrics: This indicates that while skill is present, there is still significant room for improvement across the board.

Limitations and Future Work

  • Short testing interval: The authors acknowledge that the 2-year testing interval was relatively short, particularly for evaluating rarer, larger flares (X-class).
  • Uncertainty estimation: While error bars are presented, the authors note that estimating uncertainties for skill scores in an operational context is challenging.
  • "Optimal Pth" issue: Determining an "optimal threshold" (Pth) for dichotomous metrics based on testing period information is not operationally feasible and can misrepresent performance.
  • Future work: The paper sets the stage for subsequent papers in the series that will delve into the implementation details influencing performance differences (Paper III) and explore novel methods for evaluating temporal error patterns (Paper IV). The established benchmarks will serve as a foundation for future advancements in solar flare forecasting.

Summary updated

Abstract

Abstract Solar flares are extremely energetic phenomena in our solar system. Their impulsive and often drastic radiative increases, particularly at short wavelengths, bring immediate impacts that motivate solar physics and space weather research to understand solar flares to the point of being able to forecast them. As data and algorithms improve dramatically, questions must be asked concerning how well the forecasting performs; crucially, we must ask how to rigorously measure performance in order to critically gauge any improvements. Building upon earlier-developed methodology of Paper I (Barnes et al. 2016), international representatives of regional warning centers and research facilities assembled in 2017 at the Institute for Space-Earth Environmental Research, Nagoya University, Japan to, for the first time, directly compare the performance of operational solar flare forecasting methods. Multiple quantitative evaluation metrics are employed, with the focus and discussion on evaluation methodologies given the restrictions of operational forecasting. Numerous methods performed consistently above the “no-skill” level, although which method scored top marks is decisively a function of flare event definition and the metric used; there was no single winner. Following in this paper series, we ask why the performances differ by examining implementation details (Leka et al. 2019), and then we present a novel analysis method to evaluate temporal patterns of forecasting errors in Paper IV (Park et al. 2019). With these works, this team presents a well-defined and robust methodology for evaluating solar flare forecasting methods in both research and operational frameworks and today’s performance benchmarks against which improvements and new methods may be compared.
BibTeX
@article{leka2019comparison,
  author = {K. D. Leka and Sung‐Hong Park and K. Kusano and Jesse Andries and G. Barnes and Suzy Bingham and D. Shaun Bloomfield and Aoife E. McCloskey and Véronique Delouille and D. A. Falconer and P. T. Gallagher and Manolis K. Georgoulis and Yûki Kubo and Kangjin Lee and Sang‐Woo Lee and V. V. Lobzin and JunChul Mun and Sophie A. Murray and Tarek A. M. Hamad Nageem and Rami Qahwaji and Michael Sharpe and R. A. Steenburgh and Graham Steward and Michael Terkildsen},
  title = {{A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics, and Performance Results for Operational Solar Flare Forecasting Systems}},
  journal = {The Astrophysical Journal Supplement Series},
  year = {2019},
  doi = {10.3847/1538-4365/ab2e12},
}
Figure