Abstract
Lake ice cover (LIC) is a critical climate variable for monitoring global warming and improving meteorological models. While Random Forest (RF) classifiers are widely used for mapping LIC from satellite imagery, they often lack transparency regarding prediction confidence. This study introduces a mathematically rigorous framework to estimate and decompose pixel-level uncertainty into aleatoric (data-driven) and epistemic (model-driven) components using Shannon entropy. By applying this method to MODIS data over Lake Erie, we demonstrate that rejecting the most uncertain pixels—particularly those with high epistemic uncertainty—significantly enhances the accuracy and reliability of lake ice products.
Introduction: The Need for Confidence in Climate Data
Monitoring lake ice is essential for understanding regional climate shifts and ensuring the accuracy of weather forecasts. Machine Learning (ML) models, specifically Random Forests, have become standard tools for classifying satellite pixels into ice, water, or cloud categories. However, traditional ML approaches typically provide "hard" classifications without associated error margins. For climate scientists, the lack of pixel-level uncertainty limits the ability to fuse multi-source data or integrate these maps into sensitive numerical models. This research addresses the "black box" nature of remote sensing ML by providing a method to quantify exactly how certain a model is about every pixel it classifies.
Methodology: Decomposing Uncertainty
The core of this approach lies in extracting uncertainty from the ensemble of decision trees within a Random Forest classifier. Using Shannon entropy, the researchers decompose total uncertainty (TU) into two distinct types:
- Aleatoric Uncertainty (AU): Arises from inherent noise in the data, such as sensor limitations or "mixed pixels" containing multiple surface types. It is calculated as the average entropy across individual trees.
- Epistemic Uncertainty (EU): Represents the model’s lack of knowledge about specific spectral signatures, often occurring when the model encounters data poorly represented in its training set. It is derived as the difference between total and aleatoric uncertainty ($EU = TU - AU$).
This decomposition allows researchers to distinguish between "messy data" and "model confusion," providing a more nuanced view of map reliability.
Data and Study Area
The study focused on Lake Erie, the shallowest of the Laurentian Great Lakes, which frequently experiences complete ice cover. The model utilized MODIS Terra Level 1B (MOD02) Top-of-Atmosphere (TOA) reflectance data from the winters of 2014, 2016, and 2018. To ensure global applicability, the RF was trained on an extensive dataset of over 1 million labels from various lakes above 40° latitude, collected between 2010 and 2020.
Key Insights and Results
The RF model achieved a high baseline performance with 97% accuracy on unseen test data. However, the spatial analysis of uncertainty revealed critical patterns:
- Spatial Transitions: Uncertainty is highest at the boundaries between ice, water, and clouds. Neighborhood analysis confirms that isolated pixels of one class surrounded by another are statistically more likely to be uncertain.
- The Power of Rejection: One of the most significant findings is the effectiveness of "accuracy-rejection." By discarding pixels with the highest uncertainty, the overall quality of the map improves.
- Epistemic Superiority: Epistemic uncertainty proved to be the most efficient metric for cleaning the data. Rejecting the top 10-20% of pixels based on EU yielded better accuracy gains than rejecting based on total uncertainty alone.
Conclusion and Future Directions
Quantifying uncertainty transforms "black box" ML outputs into actionable climate data. This methodology is currently under consideration for the European Space Agency’s (ESA) Climate Change Initiative (CCI+) Lakes project. While optical sensors like MODIS are limited by cloud cover, the ability to provide per-pixel confidence intervals paves the way for more sophisticated data fusion techniques, such as combining optical data with cloud-penetrating Synthetic Aperture Radar (SAR) imagery. Ultimately, these "confidence maps" ensure that climate models are built on the most trustworthy data available.
Summary updated