The Error is the Feature: How to Forecast Lightning using a Model Prediction Error

Christian Schön, Jens Dittrich, Richard MüllerKDD 2019 · DOI

TL;DR

This paper proposes a novel approach to lightning forecasting by using the prediction error of optical flow algorithms on satellite imagery as a key feature for machine learning models. This "error-as-feature" method shows promising accuracy, outperforming traditional methods and achieving over 83% accuracy for forecasts up to five hours.

What Problem Does This Paper Address?

Accurately forecasting thunderstorms and the lightning they produce remains a significant challenge in meteorology. Current methods often rely on analyzing satellite brightness temperatures or complex numerical weather prediction (NWP) models. While these approaches have improved, predicting lightning, especially for forecasts beyond one hour, is difficult, with high false alarm rates. The core problem is effectively capturing the dynamic, three-dimensional atmospheric processes that lead to convection and lightning using readily available, two-dimensional satellite data.

Methods

The paper introduces a two-stage approach. First, optical flow algorithms (specifically TV-L¹) are applied to consecutive satellite images to predict the next image in a sequence. The error between this prediction and the actual observed image is then calculated. This error is hypothesized to capture information about vertical cloud movement (convection) that 2D optical flow alone cannot directly detect.

This error, along with other features, is then fed into a second stage of machine learning models for lightning prediction. Key techniques include:

  • Optical Flow Error Feature: The absolute difference between the predicted and actual satellite image is the primary novel feature.
  • Convolutional Features: To account for spatial proximity and the movement of weather systems, convolutional operations (max, min, average, Gaussian) with various kernel sizes (3x3 to 9x9) are applied to the error maps.
  • Additional Features: Raw satellite image values, time of day, and geographic coordinates are incorporated to provide more context.
  • Machine Learning Models: Tree-based classifiers (Decision Trees, Random Forests, AdaBoost, Gradient Boosting) and a simple neural network are used for the prediction task.

Datasets

The primary dataset used is satellite imagery from the SEVIRI instrument on the Meteosat Second Generation (MSG) satellites, covering Europe and surrounding areas. This data is combined with lightning detection data from the LINET network. The study uses data from approximately one month (2018-06-01 to 2018-07-04) for training and evaluation, employing a 4-fold cross-validation strategy. The dataset is highly imbalanced, with lightning events being rare. To address this, a downsampling technique was used to balance the training data per image.

Key Insights and Ideas

  • The "Error is the Feature" Concept: The core idea that the imperfection of a predictive model (optical flow) can itself be a valuable predictor for a related phenomenon (lightning) is a significant insight. It suggests that model errors can encode subtle but important physical processes.
  • Convection as Optical Flow Error: The paper posits that the error in optical flow prediction, particularly changes in brightness, is a strong indicator of vertical cloud movement (convection), a key precursor to thunderstorms.
  • Importance of Spatial Context: Convolutional operations, especially with larger kernels (e.g., 9x9), significantly improve prediction accuracy, highlighting the need to consider the spatial context of weather phenomena.
  • Multi-channel Satellite Data: Utilizing multiple spectral channels from satellite imagery provides richer information, with channels like IR3.9 and IR9.7 appearing as particularly important features.
  • Time of Day and Location Matter: Incorporating time of day and geographic coordinates as features further enhances prediction accuracy, acknowledging diurnal cycles and geographical influences on weather.

Key Results

  • High Accuracy for Short-Term Forecasts: The best models achieved an accuracy of over 96% for predicting lightning within the next 15 minutes.
  • Sustained Accuracy for Longer Forecasts: Even for forecasts up to five hours, accuracy remained above 83%.
  • Gradient Boosting Outperforms: Gradient Boosting models generally showed the best performance among the tree classifiers, with accuracies around 91%.
  • Feature Engineering Boosts Performance: Adding non-error-based features (raw satellite values, time of day, coordinates) increased accuracy to 96.00% and improved recall and reduced false positive rates.
  • Promising ROC AUC: For a two-hour offset, the model RF2 achieved an Area Under the Curve (AUC) of 0.94, indicating strong discriminative power.

Limitations and Future Work

  • High False Positive Rate: Despite high overall accuracy, the false positive rate (nearly 6% for the best model) is still a concern for operational use, requiring further investigation.
  • Precision with Imbalanced Data: The paper acknowledges that applying the model to the original, highly imbalanced dataset would lead to a significantly lower precision (around 0.57% for a 10.62% FPR), making direct operational warnings infeasible without further improvements.
  • Data Imbalance Handling: While downsampling was used, the extreme imbalance of lightning data remains a challenge.
  • Future Work: The authors suggest exploring dimensionality reduction techniques, incorporating additional data sources (like radar or NWP models) to further improve performance and reduce false positives, and investigating neural network architectures more deeply. They also note that with more data, Random Forests could become competitive with neural networks.

Summary updated

BibTeX
@inproceedings{schon2019error,
  author = {Christian Schön and Jens Dittrich and Richard Müller},
  title = {{The Error is the Feature: How to Forecast Lightning using a Model Prediction Error}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  doi = {10.1145/3292500.3330682},
}
Figure