TL;DR
Common explainability methods (like SHAP) often identify spectral bands that align with human intuition, but this "plausibility" is a trap. In satellite imagery, high spectral redundancy means that bands labeled as "unimportant" can often train a better model than the "important" ones, revealing that current attribution methods are unstable and insufficient for feature selection.
What Problem Does This Paper Address?
In the world of eXplainable AI (XAI), we often validate a model's explanations using two informal rules:
- Convergence: Do different methods (like SHAP and Occlusion) agree on what’s important?
- Domain Consistency: Do the "important" features match what a human expert expects?
The authors argue that these are merely "sanity checks," not proofs of utility. This is especially problematic in multispectral remote sensing (satellite imagery). Because spectral bands often overlap in what they measure, a model might fixate on Band A, while the exact same information is available in Band B. If we use XAI to decide which bands to include in future, multi-million dollar satellite missions based on these attributions, we might be making a massive mistake. The paper asks: Are the features identified by XAI actually necessary for the task, or are they just one of many redundant options the model happened to pick?
Methods
The researchers evaluated three popular post-hoc attribution methods:
- GradientSHAP: A gradient-based method that approximates Shapley values.
- Shapley Sampling: A permutation-based approach to estimate feature contribution.
- Occlusion: A "leave-one-out" method that masks features and measures the drop in performance.
They applied these to a U-TAE (U-Net with Temporal Attention) model, a state-of-the-art architecture for processing satellite time-series.
Crucially, they moved beyond just looking at heatmaps. They used a protocol called ROAR (Remove and Retrain). Instead of just masking bands on a "frozen" model, they retrained new models from scratch on specific subsets of bands (e.g., the "Top-5" vs. the "Bottom-5") to see if the XAI rankings actually predicted which bands were most useful for learning.
Datasets
The study utilized the PASTIS benchmark, a large-scale dataset for agricultural crop-type classification in France. It consists of:
- Sentinel-2 satellite time-series: 10 spectral bands (visible, red-edge, NIR, and SWIR).
- Multi-temporal data: Between 38 and 61 observations over a season.
- Labels: 18 crop classes (though the authors focused on a 7-class subset for diagnostic stability).
Key Insights and Ideas
- The "Identical Run" Paradox: Perhaps the most startling finding is that two training runs with the exact same code and seed can reach the same accuracy but yield different attribution rankings. In "Run A," the bottom-ranked bands retrained better than the top-ranked ones. In "Run B," the result flipped. This suggests XAI rankings are highly sensitive to the specific "solution" a non-deterministic training process happens to find.
- Plausibility $\neq$ Necessity: All methods correctly identified "vegetation-sensitive" bands (like Near-Infrared), which matches remote sensing theory. However, retraining showed that these "plausible" bands weren't uniquely necessary.
- Spectral Redundancy is a "Hidden Variable": The authors used a metric called the Participation Ratio to show that the "effective dimensionality" of a 5-band subset is often as low as 1.35. Because the bands are so correlated, the model has a "choice" of which band to rely on; XAI simply points to the one it chose, not the only one that works.
- Learners Disagree: Bands that are important for a simple pixel-level classifier are often dispensable for a complex deep learning model (like U-TAE) that can also exploit spatial and temporal context.
Key Results
- Retraining Failure: In one primary experiment (Run A), models trained on the five lowest-ranked bands achieved a mean IoU of 0.9047, significantly outperforming models trained on the five highest-ranked bands (0.8893).
- Low Correlation: At the per-class level, different XAI methods showed poor agreement, with Spearman rank correlations as low as 0.09.
- Redundancy Quantified: Analysis of all 252 possible five-band subsets showed that they all reconstruct the majority of the spectral variance, explaining why so many different subsets perform similarly despite having vastly different XAI rankings.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The study focused on a single model architecture (U-TAE) and one specific dataset (PASTIS/Sentinel-2). It is unclear if these findings hold for other sensors (like hyperspectral) or different geographic regions.
- Attribution Type: The study focused on spectral attribution. Temporal and spatial attribution (which time steps or pixels are most important) were not explored.
- Feature Selection: While the paper critiques using XAI for feature selection, it doesn't propose a "perfect" alternative, though it suggests that retraining on subsets is a necessary "cheap" additional check.
The takeaway for practitioners is clear: Don't trust a single attribution map. If you are using XAI to make high-stakes decisions about feature importance, you must test those rankings through retraining and check for stability across multiple model runs.
Summary updated