Satellite and aerial sensors record many more colors than a camera does: near-infrared, red-edge, shortwave infrared, and atmospheric bands, often 10 to 13 channels per pixel. A popular way to learn from this imagery without labels is contrastive learning, which trains an encoder to recognize that two different "views" of the same scene belong together and that views of different scenes do not. When the two views are different groups of spectral bands of the same patch (visible light in one view, non-visible bands in the other), the model has to find what the band groups share. Nothing forces it to find the meaningful shared content. It can match views through whichever one or two bands make the matching game easiest. We propose to watch, for every training sample, which input band the matching decision currently depends on most, measured by backpropagating the contrastive score to the input bands. We then temporarily hide that band. This pushes the encoder to spread its evidence across the spectrum instead of settling on a spectral shortcut.
In two sentences: the gradient of "how much more does this view match its partner than the impostors" with respect to each spectral band tells us which band the encoder is exploiting on that sample, and masking that band during training is a targeted, cheap way to break shortcut solutions. The measurable promise is better label-efficient transfer and much less damage when bands go missing at test time. The decisive control is plain random band dropout, which the method must clearly beat.
Core Claim
In cross-spectral contrastive pretraining, the bands that make instance discrimination easy are not necessarily the bands that carry land-cover semantics. A per-sample, per-band attribution of the contrastive margin exposes this, and adversarially masking the top-attributed band yields encoders that are more robust and more transferable than encoders trained with random band dropout at the same masking rate.
The claim rests on a tension the literature leaves open. Can contrastive learning avoid shortcut solutions? shows that contrastive learners latch onto easy features and suppress harder ones, and that lowering the temperature merely shifts the bias. Multispectral data is full of exactly such easy, high-contrast signals: water absorbs near-infrared, vegetation reflects it, and haze shows up coherently across bands. Nobody has checked whether multispectral contrastive encoders learn biophysical structure or ride these shortcuts, and no method acts on the answer during training.
What Exists and How This Differs
Three families of prior work touch this question, and none closes it.
The first family is multispectral contrastive learning, which supplies the training setting.
- SACo+ groups Sentinel-2 bands into seven hand-chosen functional categories plus texture descriptors, and aligns them to a shared anchor.
- Viewmaker networks for multispectral data learn an adversarial perturbation of the whole band tensor under a distortion budget.
- PIMC reports the strongest EuroSAT numbers in the file.
All three treat bands as fixed inputs or fixed groups. None measures which band a given sample's alignment depends on. Viewmaker is the closest in spirit, since it is also an adversary acting on views. Its adversary, however, maximizes loss through a bounded perturbation spread over all channels. It is not told which band the encoder relies on, and it cannot remove a band outright.
The second family explains decisions made on multispectral inputs. A Comparison of Deep Saliency Map Generators on Multispectral Data in Object Detection evaluates Grad-CAM, RISE and SIDU per spectrum on a supervised detector. The color out of space uses Grad-CAM qualitatively to show that its spectral branch attends to materials and its RGB branch to shapes. Both motivate the proposal: attributions across spectra differ in informative ways. But both work post hoc, both produce spatial maps rather than a ranking over bands, and neither involves a contrastive objective.
The third family supplies the method: gradient attribution adapted to contrastive or confusable targets.
- COCOA attributes the similarity of a representation to a "corpus" minus its similarity to a "foil" set.
- CASE shows that Grad-CAM produces nearly the same map for different classes. It fixes this by removing the component of the target gradient that aligns with the gradients of confusable classes.
- Logit Attribution Matching uses attributions as a training signal for domain generalization. - Saliency Guided Contrastive Learning uses graph-cut saliency to steer spatial crops.
All of these operate on the spatial axis of RGB or medical images. We borrow two ideas from them: the corpus-versus-foil target from COCOA and the orthogonal projection from CASE. Our changes are to attribute along the band axis instead of over pixels, to use the positive pair and the in-batch negatives as the corpus and foil (so no manual reference sets are needed), and to feed the result back into training.
Spectral Occlusion constrains the design. It argues that spatial heatmaps cannot express globally distributed properties. So we aggregate attributions over space into one score per band rather than trying to read band importance off a heatmap.
Mechanism
Let $x$ be an image patch. Let $v_1(x)$ be its visible-band view and $v_2(x)$ its non-visible-band view, with encoders producing unit vectors $z_1, z_2$ and in-batch negatives $z_2^{-}$. We define the contrastive margin for the sample as
$$ m(x) = z_1^\top z_2 - \log \sum_{j} \exp\left(z_1^\top z_{2,j}^{-}/\tau\right)\tau , $$
that is, positive similarity minus a soft maximum over negative similarities at temperature $\tau$. This is the quantity the InfoNCE loss increases.
Following CASE, we write $g = \partial m/\partial v_2$ and take $\bar g^{-}$ to be the mean gradient of the negative-similarity terms alone. We remove the component of $g$ along $\bar g^{-}$, keeping only evidence specific to this pair rather than generic "looks like any Sentinel-2 tile" evidence. The band score is the input-times-gradient mass of the projected gradient $\tilde g$, summed over pixels:
$$ a_b(x) = \sum_{i,j} \left| v_{2}(x)_{b,i,j}\, \tilde g_{b,i,j} \right| , $$
where $b$ indexes bands and $(i,j)$ indexes pixels.
At each step, with probability $p$, the band $\arg\max_b a_b(x)$ is replaced by its dataset mean for that sample, in both the input and the loss. The same is done symmetrically for the visible view. Scores are computed from the previous step's parameters, which adds one backward pass per step.
We will verify that $a_b$ is faithful before relying on it. The test adapts insertion and deletion curves from Bayer et al. to the band axis and measures rank correlation with exact band occlusion, which on 13 bands costs 13 forward passes and is affordable offline.
Research Program
The work has three stages, and each is useful even if the next fails.
Stage 1: faithfulness. We establish whether projected contrastive-margin gradients rank bands faithfully, using exact band occlusion as ground truth on held-out patches. The projection is a design choice here. If raw gradients are as faithful, we drop the projection.
Stage 2: shortcut audit. We train standard cross-spectral contrastive encoders (visible versus non-visible views, and on So2Sat, radar versus optical) and aggregate the attributions per land-cover class. The aim is to test whether matching depends on bands with little semantic content, such as the 60-meter aerosol, water-vapour and cirrus bands. This is a hypothesis, not a known result.
The per-class band profiles also give a label-free band-importance ranking. We can compare it against the supervised findings in EuroSAT, where red-edge and shortwave infrared beat standard near-infrared on specific classes, and in the Surrogate Contrastive Network, where NIR1 was most informative and Coastal Blue least informative for buildings.
Stage 3: saliency-guided masking. We train with saliency-guided band masking and compare it against the baselines below.
Baselines
The naive reference point is the same contrastive setup with no band masking: MoCo-v2-style InfoNCE between the visible and non-visible views, same ResNet-18, same augmentations. For context, SACo+ reports 83.72% EuroSAT top-1 for MoCo v2. We will re-measure under our own protocol rather than borrow that figure.
The ablation that isolates the key ingredient is random band dropout at exactly the matched masking rate $p$. It is the same pipeline with the argmax replaced by a uniform draw. If saliency-guided masking does not beat it, the gradient signal adds nothing and the paper's claim is false.
A second ablation replaces gradient scores with exact occlusion scores at the same masking rate. It tests whether gradients are merely a cheap approximation (fine) or a worse one (then occlusion should be preferred, and we would report that).
The strongest existing methods are PIMC and Viewmaker. PIMC reports 97.44% EuroSAT accuracy with a ResNet-18 image encoder, against 96.10% for DINO-MC and 93.33% for SeCo. Viewmaker reports 96.40% on EuroSAT and 60.08% (Sentinel-2) and 38.25% (Sentinel-1) on So2Sat. Viewmaker is the most relevant competitor because it is also an adversary on multispectral views, so it tests whether a learned perturbation already does what targeted band removal does. We will also run SACo+ (94.72% EuroSAT) as the representative of hand-assigned band groups.
Every method will be pretrained on the same unlabeled data with the same backbone, epochs and hardware budget, including our extra backward pass, which we count against us. Each method gets the same number of hyperparameter trials, and downstream probes are identical. Beating the baselines means two things. First, a statistically reliable gain over random band dropout and over Viewmaker on low-label BigEarthNet and on So2Sat. Second, a clearly smaller accuracy drop under test-time missing bands. Matching PIMC on full-label EuroSAT, where numbers are near ceiling, is not required.
Why not just…?
Why not just use random band dropout? This is the real threat, and the experiment is built around it. The argument for targeting is arithmetic. If an encoder leans on one of 13 bands, random single-band dropout removes that band only about one time in thirteen. Targeted masking removes it whenever it dominates, which is what the shortcut literature suggests is needed to surface suppressed features (Robinson et al.). If random dropout matches us at equal rate and compute, we will report that the simpler method suffices.
Why not just occlude each band and measure the drop, instead of using gradients? Band-level occlusion is cheap for evaluation, and we use it as ground truth. During training, though, it costs 13 extra forward passes per sample per step, against one backward pass for gradients. The 320-fold gap between Grad-CAM and RISE in Bayer et al. concerns spatial perturbation, so we do not claim it transfers. The occlusion-guided ablation settles the question directly: if it wins by a margin worth its cost, it should replace gradients.
Why not just learn a gate on the input bands, or select bands once up front? A learned per-band gate trained through the contrastive loss would amplify whatever band makes matching easiest, which is the shortcut itself. Static selection, as in the Surrogate Contrastive Network, needs labels and yields one global subset. That conflicts with the per-class variation reported in EuroSAT and Agriculture-Vision. We include a gated variant in the audit to show whether it concentrates weight on the bands our attributions flag.
Why not just train supervised on all stacked bands? Labels are scarce. The color out of space reports 0.486 mAP from scratch against 0.622 mAP with self-supervised pretraining at 1,000 BigEarthNet labels. We include the from-scratch model in the low-label comparison as a floor.
Proposed Experiments
Pretraining uses unlabeled BigEarthNet Sentinel-2 patches for the optical setting and So2Sat LCZ42 radar-optical pairs for the genuinely cross-modal setting. Evaluation uses:
- EuroSAT linear probe, with the cirrus band excluded if needed to match the 12 BigEarthNet bands;
- BigEarthNet multi-label mAP with 1,000 labels, where the 0.622 mAP reported for colorization pretraining in The color out of space is the external reference;
- So2Sat accuracy per sensor.
Robustness is measured by dropping each band, and each semantically grouped set of bands, at test time. We report the accuracy drop, following the missing-modality framing of GMC.
A controlled shortcut test provides ground truth. We append to both views a synthetic channel carrying a per-image random code, which makes instance matching trivial. An unmasked encoder should collapse onto this channel, visible as degraded probe accuracy. We then check two things: whether the attribution ranks the planted channel first, and whether guided masking recovers probe accuracy faster than random dropout at equal rate.
Agriculture-Vision provides a small-band-count sanity check. With only four channels (NRGB), where near-infrared lifts segmentation mIoU from 39.63% to 43.66% (Agriculture-Vision), we test whether guided masking keeps the near-infrared benefit while reducing collapse when near-infrared is removed.
The decisive checks, in order, are:
- Band attributions correlate with exact occlusion on held-out data. If they do not, the method has no foundation.
- On the planted-shortcut test, attribution finds the planted channel and guided masking beats random dropout.
- On the natural data, guided masking beats random dropout and Viewmaker on low-label BigEarthNet, on So2Sat, and on missing-band robustness, at matched compute.
A null result on the third check after passing the first two would still be informative. It would mean real Sentinel-2 contrastive encoders are not shortcut-dominated along the band axis, which the audit in Stage 2 would confirm or refute directly.
References
- Can contrastive learning avoid shortcut solutions?
- Semantically-Aware Contrastive Learning for multispectral remote sensing images
- Multispectral Contrastive Learning with Viewmaker Networks
- Pixel-Wise Multimodal Contrastive Learning for Remote Sensing Images
- A Comparison of Deep Saliency Map Generators on Multispectral Data in Object Detection
- The color out of space: learning self-supervised representations for Earth Observation imagery
- Contrastive Corpus Attribution for Explaining Representations
- CASE: Contrastive Activation for Saliency Estimation
- Contrastive Domain Generalization via Logit Attribution Matching
- Saliency Guided Contrastive Learning on Scene Images
- Spectral Occlusion - Attribution Beyond Spatial Relevance Heatmaps
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
- Surrogate Contrastive Network for Supervised Band Selection in Multispectral Computer Vision Tasks
- Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis
- Geometric Multimodal Contrastive Representation Learning