Illusion of Explainability: A Sanity Check for Spectral Band Attribution in Satellite Time-Series Crop Classification

Sumit Tahiliani, Avi Mathur, Sandarbh Agarwal, Rakshith Sathish, Ashwathi NambiararXiv 2026 · arXiv

TL;DR

Common explainability methods (like SHAP) often identify spectral bands that align with human intuition, but this "plausibility" is a trap. In satellite imagery, high spectral redundancy means that bands labeled as "unimportant" can often train a better model than the "important" ones, revealing that current attribution methods are unstable and insufficient for feature selection.

What Problem Does This Paper Address?

In the world of eXplainable AI (XAI), we often validate a model's explanations using two informal rules:

  1. Convergence: Do different methods (like SHAP and Occlusion) agree on what’s important?
  2. Domain Consistency: Do the "important" features match what a human expert expects?

The authors argue that these are merely "sanity checks," not proofs of utility. This is especially problematic in multispectral remote sensing (satellite imagery). Because spectral bands often overlap in what they measure, a model might fixate on Band A, while the exact same information is available in Band B. If we use XAI to decide which bands to include in future, multi-million dollar satellite missions based on these attributions, we might be making a massive mistake. The paper asks: Are the features identified by XAI actually necessary for the task, or are they just one of many redundant options the model happened to pick?

Methods

The researchers evaluated three popular post-hoc attribution methods:

  • GradientSHAP: A gradient-based method that approximates Shapley values.
  • Shapley Sampling: A permutation-based approach to estimate feature contribution.
  • Occlusion: A "leave-one-out" method that masks features and measures the drop in performance.

They applied these to a U-TAE (U-Net with Temporal Attention) model, a state-of-the-art architecture for processing satellite time-series.

Crucially, they moved beyond just looking at heatmaps. They used a protocol called ROAR (Remove and Retrain). Instead of just masking bands on a "frozen" model, they retrained new models from scratch on specific subsets of bands (e.g., the "Top-5" vs. the "Bottom-5") to see if the XAI rankings actually predicted which bands were most useful for learning.

Datasets

The study utilized the PASTIS benchmark, a large-scale dataset for agricultural crop-type classification in France. It consists of:

  • Sentinel-2 satellite time-series: 10 spectral bands (visible, red-edge, NIR, and SWIR).
  • Multi-temporal data: Between 38 and 61 observations over a season.
  • Labels: 18 crop classes (though the authors focused on a 7-class subset for diagnostic stability).

Key Insights and Ideas

  • The "Identical Run" Paradox: Perhaps the most startling finding is that two training runs with the exact same code and seed can reach the same accuracy but yield different attribution rankings. In "Run A," the bottom-ranked bands retrained better than the top-ranked ones. In "Run B," the result flipped. This suggests XAI rankings are highly sensitive to the specific "solution" a non-deterministic training process happens to find.
  • Plausibility $\neq$ Necessity: All methods correctly identified "vegetation-sensitive" bands (like Near-Infrared), which matches remote sensing theory. However, retraining showed that these "plausible" bands weren't uniquely necessary.
  • Spectral Redundancy is a "Hidden Variable": The authors used a metric called the Participation Ratio to show that the "effective dimensionality" of a 5-band subset is often as low as 1.35. Because the bands are so correlated, the model has a "choice" of which band to rely on; XAI simply points to the one it chose, not the only one that works.
  • Learners Disagree: Bands that are important for a simple pixel-level classifier are often dispensable for a complex deep learning model (like U-TAE) that can also exploit spatial and temporal context.

Key Results

  • Retraining Failure: In one primary experiment (Run A), models trained on the five lowest-ranked bands achieved a mean IoU of 0.9047, significantly outperforming models trained on the five highest-ranked bands (0.8893).
  • Low Correlation: At the per-class level, different XAI methods showed poor agreement, with Spearman rank correlations as low as 0.09.
  • Redundancy Quantified: Analysis of all 252 possible five-band subsets showed that they all reconstruct the majority of the spectral variance, explaining why so many different subsets perform similarly despite having vastly different XAI rankings.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study focused on a single model architecture (U-TAE) and one specific dataset (PASTIS/Sentinel-2). It is unclear if these findings hold for other sensors (like hyperspectral) or different geographic regions.
  • Attribution Type: The study focused on spectral attribution. Temporal and spatial attribution (which time steps or pixels are most important) were not explored.
  • Feature Selection: While the paper critiques using XAI for feature selection, it doesn't propose a "perfect" alternative, though it suggests that retraining on subsets is a necessary "cheap" additional check.

The takeaway for practitioners is clear: Don't trust a single attribution map. If you are using XAI to make high-stakes decisions about feature importance, you must test those rankings through retraining and check for stability across multiple model runs.

Summary updated

Abstract

Attribution methods are commonly validated through convergence across methods and consistency with domain knowledge, yet neither criterion establishes whether identified features are necessary for a task. This limitation is particularly relevant in multispectral remote sensing, where overlapping spectral information makes individual bands potentially substitutable. We investigate this gap in crop-type classification using GradientSHAP, Shapley sampling, and Occlusion on a U-TAE segmentation model trained on Sentinel-2 time series from the PASTIS benchmark. All three methods identify vegetation-sensitive spectral regions consistent with domain knowledge, while Shapley and Occlusion exhibit strong ordering sensitivity in deletion and insertion tests. However, these properties do not reliably predict retraining performance. Aggregate attribution consensus masks substantial per-class disagreement, with pairwise mean Spearman correlations ranging from 0.09 to 0.79. Models retrained from scratch using the five lowest-ranked bands perform comparably to or better than those using the five highest-ranked bands. Furthermore, attribution rankings from an independently trained model with identical configuration and seed identify a different top-five subset that yields better retraining performance. Analysis of all 252 five-band subsets reveals consistently low effective dimensionality (1.35 to 2.62), suggesting substantial spectral redundancy. These findings demonstrate that plausible and apparently faithful attribution rankings do not necessarily identify task-relevant bands for improved retrained performance, highlighting the limitations of band-level feature attribution in spectrally redundant input spaces.
BibTeX
@misc{tahiliani2026illusion,
  author = {Sumit Tahiliani and Avi Mathur and Sandarbh Agarwal and Rakshith Sathish and Ashwathi Nambiar},
  title = {{Illusion of Explainability: A Sanity Check for Spectral Band Attribution in Satellite Time-Series Crop Classification}},
  year = {2026},
  eprint = {2611.19889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19889},
}
Figure