Models that estimate forest biomass or map floods from satellites combine a year of radar and optical images of the same place. Much of that stack is useless at any given pixel. A July optical image may be clear over one corner of a tile and covered by cloud, haze or cloud shadow over the rest, and a radar image may be degraded by speckle or terrain distortion. The current strongest models collapse the time series with softmax attention, which weights each acquisition date by a normalized score. Softmax has to give every date some positive weight, so corrupted observations always leak into the fused representation. The usual fix for bad inputs is channel reweighting in the style of Squeeze-and-Excitation, which averages each feature map over the whole image to decide how much to trust it. That fix averages clean and cloudy pixels together, so it cannot represent a cloud that covers only part of an image.
We propose one small module that addresses both problems. For every pixel, sensor and date, it estimates a reliability score from the local neighbourhood at several spatial scales. It then lets the temporal attention abstain: when all dates at a pixel look unreliable, weight goes to a learned "none of the above" slot instead of being spread across corrupted observations. The claim can be tested directly. If learned, spatially local reliability gating is the missing ingredient, it should beat both a well-tuned softmax temporal encoder and the obvious hand-crafted alternative, masking pixels flagged as cloudy. The advantage should widen as synthetic corruption increases, and it should appear on corruptions that no cloud mask flags.
Core Claim
In multi-temporal multi-sensor fusion, the errors attributed to "noisy sensors" come mostly from two design choices. The first is that temporal attention cannot assign zero weight. The second is that channel gating decides trust at the level of the whole image rather than the pixel. Replacing both with a pixel-level, multi-scale reliability gate that can drive a sensor-date's weight to zero will reduce biomass RMSE on BioMassters and will degrade more gracefully under partial occlusion than masking, sparse attention, or spatially aware channel gating alone.
Where the idea comes from and how it differs
The motivation comes from the time-series literature. There, unconstrained softmax attention over long sequences has been shown to spread weight nearly uniformly into a "trivial representation" (DeformableTST). It also overreacts to isolated noisy points (Li et al.; CrossGNN) and picks up spurious long-range associations (TimeBridge; DeCoP). These papers study forecasting on one regular sensor stream and fix the problem through segmentation, locality or decomposition. None considers a stack of heterogeneous sensors in which whole observations are sometimes invalid.
Video moment retrieval is the closest analogue. DRNet generates purification masks before attention because over 70% of clips are background, and removing its denoising module costs 11.71% performance. That supports "purify before attending." DRNet, however, needs a text query and an auxiliary reconstruction loss. Our gate is driven only by the sensor data and needs no extra objective.
On the channel side, the method draws on Squeeze-and-Excitation as used for multi-sensor structural monitoring (Informer-SEnet) and 12-lead ECG (SE-ResNet). Both show that learned per-channel trust helps when some sensors are noisy. Both also squeeze globally, so a lead or strain gauge is trusted or distrusted as a whole. That is adequate for a 1-D sensor, but it fails for an image band that is half cloud.
DFYP is the closest existing combination. It applies spectral channel attention inside a dynamic fusion of MODIS and Sentinel-2 for yield prediction. Our proposal differs in two ways. First, reliability in our module is spatially local and specific to each date, not a spectral weight per channel. Second, it acts on the temporal attention's normalization by allowing abstention, rather than rescaling features before a standard fusion.
From multi-scale spatial attention we reuse one technique: parallel receptive fields combined by a learned per-pixel selector. This follows the parallel-kernel design in AFCN for SAR segmentation and the dilated pyramids in DPSDA-Net. Here the selector chooses the neighbourhood size over which trust is estimated, not which features to segment with. Scale matters because cloud cover is judged well from a small window, while radar speckle and layover need wider context.
The fusion literature supplies the setting and a warning. BioMassters fuses 12 months of Sentinel-1 and Sentinel-2. Its strongest baseline is U-TAE, a U-Net that collapses the time series with temporal attention, at 27.49 t/px RMSE. Multi3Net and the multi-input landslide U-Net show large gains from adding radar to optical data. That tells us radar is genuinely complementary, so the gate must not simply learn to prefer one sensor. Known saturation of optical and C-band signals in dense canopy limits how much any fusion method can gain there. We therefore report errors stratified by biomass level rather than claim uniform improvement.
Mechanism
Let $x_{m,t}$ be the encoder feature map for sensor $m$ at date $t$. At each pixel $p$ and for each window size $k \in \{3, 9, 27\}$, we compute a local squeeze $z^{(k)}_{m,t}(p)$. This is the mean and variance of the features within the window, a local analogue of global average pooling. A small per-pixel softmax over the three scales combines them into one descriptor. A two-layer bottleneck with a sigmoid then maps that descriptor to a reliability $g_{m,t}(p) \in [0,1]$.
The temporal attention score $s_{m,t}(p)$ comes from U-TAE's own query–key product. The weight on each sensor-date becomes
$$a_{m,t}(p) = \frac{g_{m,t}(p)\, e^{s_{m,t}(p)}}{e^{b(p)} + \sum_{m',t'} g_{m',t'}(p)\, e^{s_{m',t'}(p)}},$$
where $b(p)$ is a learned per-pixel abstention logit. The remaining mass $1 - \sum a$ goes to a learned null embedding. As a result, a sensor-date with $g \to 0$ receives exactly vanishing weight, and a pixel whose observations are all poor falls back to a prior instead of averaging noise.
The gate is trained end-to-end on the regression loss alone. The only extra term is an optional weak entropy penalty on the scale selector. We deliberately use no cloud-mask supervision, so that we can test whether the gate learns reliability that masks miss.
Research Program
We will implement the module as a drop-in replacement for the temporal attention in U-TAE. The encoder, decoder and training recipe stay the same, which isolates the effect of the gate.
Three analyses accompany the accuracy numbers. The first compares the learned $g$ for optical dates against cloud-probability layers where available, and against visually annotated haze and shadow patches, to see whether the gate tracks physical occlusion. The second examines radar gates as a function of local slope, to test whether the module learns to distrust layover and shadow regions without being told. The third maps where the abstention slot fires and correlates that map with error, to check whether abstention marks genuinely uninformative pixels.
Baselines
The naive reference point is U-TAE exactly as reported in BioMassters, with RMSE 27.49 t/px. We will re-run it with our own data pipeline and seeds, because any gain must be measured against our reproduction rather than the published figure. The same comparison includes an even simpler reference: the per-pixel median over time of each band, fed to the same U-Net without temporal attention.
The strongest existing method is that U-TAE baseline, retuned with the same hyperparameter search budget as our model. Both models get an equal number of trials over learning rate, weight decay and attention heads, the same epochs, and five seeds each. This matters because it is the model the benchmark authors found best and the one our module modifies.
The strongest hand-crafted competitor is U-TAE with hard masking. Here, optical pixels flagged cloudy by the provided cloud information (or an s2cloudless-style threshold if none is provided) receive a score of $-\infty$ before softmax. This is the baseline a practitioner would build first.
Three ablations isolate the ingredients:
- Global squeeze: reliability from the whole-image squeeze, i.e. Squeeze-and-Excitation applied per sensor-date as in Informer-SEnet.
- Single scale: a single fixed 9×9 window with no scale selector.
- No abstention: the gate applied but with $b \to -\infty$.
For cross-sensor channel attention we will also run a DFYP-style spectral channel attention placed before U-TAE's temporal encoder, since DFYP is the closest combination in the literature. The file reports no BioMassters number for DFYP.
Our method counts as beating a baseline only if it satisfies two conditions. Its mean RMSE across five seeds must be lower by more than twice the seed standard deviation. It must also win or tie on the high-biomass stratum rather than only in easy, low-biomass pixels.
Why not just…?
Why not just mask cloudy pixels before attention? This is the most serious threat. If most of the damage comes from thick cloud that standard masks flag, hard masking may match our gate at no cost. The experiments settle it in three places. The first is the direct comparison with the masked U-TAE baseline. The second is a stratified evaluation on pixels whose dates contain haze, thin cirrus or cloud shadow that the mask does not flag. The third is radar corruption, which no optical cloud mask can see. If the gate does not beat masking on those strata, we will report that the benefit reduces to cloud masking and drop the claim.
Why not just use sparse attention such as sparsemax, which already allows exact zeros? Sparse normalization zeroes low-scoring dates. The score, however, comes from a query–key product on features that clouds have already contaminated, so a bright cloud can score high. Our gate is computed from local feature statistics separately from the score. The "sparsemax U-TAE" run, with the same tuning budget, tests whether that separation matters. If sparsemax matches us, abstention rather than reliability estimation is the active ingredient, and the paper should say so.
Why not just add spatial context to Squeeze-and-Excitation and keep softmax? A spatially aware channel gate, such as our single-scale gate without abstention, is cheaper and may capture most of the gain. The "no abstention" and "single scale" ablations measure exactly this. We expect abstention to matter most for pixels where every optical date is poor, and we report that subset separately.
Why not just train a bigger U-TAE with cloud augmentation? Capacity and augmentation can teach a softmax encoder to down-weight clouds, but softmax still cannot assign zero. We will train a U-TAE with parameter count matched to our model, using the same synthetic-cloud augmentation, as an additional control.
Proposed Experiments
The primary benchmark is BioMassters, scored by RMSE in t/px against the reproduced 27.49 t/px U-TAE baseline, with errors stratified by biomass decile.
The decisive check is a controlled corruption sweep on the test set. We will select optical dates that the cloud information marks clear. We then paste real cloud and shadow patches cut from other tiles at 10–70% spatial coverage over 0–8 of the 12 months. Separately, we inject multiplicative speckle into radar at increasing strength. We plot RMSE against corruption level for every baseline. The proposal predicts two things: a flatter curve than softmax U-TAE and masked U-TAE, and a gap that grows with partial coverage specifically. Partial coverage is the regime where global squeezing fails by construction. If the curves are parallel, the claim is wrong.
For transfer, we use AGBD. It fuses Sentinel-2, PALSAR-2, DEM and GEDI labels across 11 regions, and its full-fusion ResNet baseline reaches RMSE 57.14 Mg/ha. Because AGBD inputs are not a long time series in the file's description, we test only the spatial reliability gate across sensors there, with leave-one-region-out evaluation. This asks whether learned local trust generalizes across biomes, where BioMassters-style boreal training is known to transfer poorly.
A third check uses the Sentinel-1 plus Sentinel-2 rapid-response setting of Multi3Net, reported at 59.7% mIoU on Hurricane Harvey. Storm-season optical imagery there is heavily clouded, which tests whether the gate shifts trust to radar when it should.
Across all settings, compute overhead is reported as parameters and wall-clock time. The three-scale squeeze adds depthwise pooling, not new convolution branches, which avoids the parameter redundancy noted for DPSDA-Net.
References
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement
- TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecasting
- DeCoP: Enhancing Self-Supervised Time Series Representation with Dependency Controlled Pre-training
- Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrieval
- Research on Long-Term Structural Response Time-Series Prediction Method Based on the Informer-SEnet Model
- Classification of Cardiac Abnormalities From ECG Signals Using SE-ResNet
- DFYP: A Dynamic Fusion Framework with Spectral Channel Attention and Adaptive Operator learning for Crop Yield Prediction
- A Novel Attention Fully Convolutional Network Method for Synthetic Aperture Radar Image Segmentation
- DPSDA-Net: Dual-Path Convolutional Neural Network with Strip Dilated Attention Module for Road Extraction from High-Resolution Remote Sensing Images
- BioMassters: A Benchmark Dataset for Forest Biomass Estimation using Multi-modal Satellite Time-series
- Multi3Net: Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery
- A Multi-Input Channel U-Net Landslide Detection Method Fusing SAR Multisource Remote Sensing Data
- AGBD: A Global-scale Biomass Dataset