Transformers restore images by letting every patch look at other patches and borrow information from them. This lookup, called self-attention, costs time and memory that grow with the square of the number of patches. That cost is the main obstacle for large inputs such as hyperspectral images, which are camera cubes with dozens to hundreds of wavelength bands. The standard fixes are to restrict each patch to a local window, or to drop patches judged unimportant. Both fixes misbehave in the same situation. When a sensor produces stripes, dead lines, or impulse spikes, the corrupted patches look high-contrast and "salient", so windowed attention mixes them into clean neighbours, and pruning methods keep them. We propose to reverse the logic. A small learned noise model, a mixture of Gaussian components fitted to the corruption, estimates how unreliable each patch and each band is. Unreliable patches are then removed from the set of patches that others read from, while every patch is still restored. Dropping unreliable readers from the source set is where the compute saving comes from.
Put in two sentences: attention should read less from positions that a fitted noise model says are corrupted, and those positions can be dropped from the key side of attention, which is cheaper than dropping them outright and more correct. We test whether this one change gives better denoising under structured sensor noise at lower cost than windowed and pruned hyperspectral Transformers.
Core Claim
Transformer with a Mixture of Gaussian Keys showed that softmax attention equals the posterior of a Gaussian mixture in which each key is a cluster mean. That paper used the view only to enrich internal key representations in clean text. We supply the method it did not: if key $k_j$ is observed through corruption with variance $\sigma_j^2$, the correct posterior widens that component. The attention weight of query $q$ on key $j$ becomes
$$ a_{ij} \propto \exp\!\left(-\frac{\lVert q_i - k_j\rVert^2}{2(\tau^2+\sigma_j^2)} - \frac{d}{2}\log(\tau^2+\sigma_j^2)\right), $$
where $\tau^2$ is the base attention temperature and $d$ is the key dimension. The second term shrinks the weight of noisy keys regardless of how similar they look. The per-key variance comes from a learned noise mixture with $C$ components of variance $s_c^2$. For each pixel, or each band of a pixel, we compute responsibilities $r_{jc}$, the posterior probability that its residual came from component $c$, and set $\sigma_j^2=\sum_c r_{jc}s_c^2$.
Structured corruption is not a single Gaussian. Impulse noise, stripes, and thermal noise have very different spreads, which is the lesson of Robust Principal Component Analysis with Complex Noise and Denoising Hyperspectral Image with Non-i.i.d. Noise Structure. A mixture is therefore needed to push responsibilities toward near-binary "this entry is garbage" decisions.
Keys with $\sigma_j^2$ above a threshold contribute almost nothing, so we drop them. The cost falls from $N^2$ to $N\cdot N_k$, where $N_k$ is the number of reliable keys. In the spectral attention path the same rule removes corrupted bands from the source set, and the band-specific noise parameters follow NMoG-LRMF's non-i.i.d. structure. In linear-attention variants, the per-key weight simply scales each term in the key-value sum, so the cost stays linear.
Research Program
We build on a spatial-spectral hyperspectral backbone in the style of SERT, which uses rectangular windows and a low-rank memory, and SST, which pairs windowed spatial attention with global spectral attention. We change only the attention weighting and the key set. A light head predicts a coarse clean estimate. From the residual, it computes mixture responsibilities in closed form, using a per-band mixture with $C$ around 3 to 5 components. On synthetic training data, where the true noise is known, the head is trained with the mixture negative log-likelihood of the true noise alongside the usual restoration loss. On real data it is fitted to its own residuals.
This differs from DNA-Net in two ways. DNA-Net unrolls an optimisation with a fixed $L_1$-plus-Frobenius noise model and predicts regularizer weights, and it never lets the noise model change what attention reads. Its unrolled stages also add latency, whereas our change is feed-forward.
It also differs from Training Noise Token Pruning and SG-Former. Both score token importance on clean ImageNet inputs, and TNT's Gaussian noise is only an optimisation device for making pruning differentiable. In our method the noise is the physical corruption being modelled. Pruning acts on keys, never on queries, so corrupted pixels are still restored.
The analysis has three parts. First, check that the predicted $\sigma_j^2$ tracks the ground-truth corruption masks available in synthetic noise. Second, measure how many keys can be dropped at each noise level. Third, test whether the log-variance term in the equation matters beyond simple rescaling.
Baselines
The naive reference is the unmodified backbone, SERT-style, trained with the same data, schedule, and loss, with full key sets inside its windows. This isolates what the noise-aware keys add. SERT reports 1018.9 GFLOPs against 3652.8 for Restormer, and more than 0.9 dB PSNR over QRNN3D on ICVL complex noise. The file gives no absolute ICVL PSNR, so we will retrain it ourselves.
The strongest existing competitor on accuracy is DNA-Net, which reports an average 1.49 dB PSNR gain over SERT for its 5-iteration variant and about 0.09 s per image for its 2-iteration variant. We run both variants with their released settings on the same splits. Beating DNA-Net means matching the PSNR of DNA-Net-B while running at or below DNA-Net-L's runtime, or exceeding DNA-Net-L at equal runtime.
The classical model-based reference is NMoG-LRMF, which reports 50.38 dB mean PSNR on Washington DC Mall under mixture noise, against 43.22 dB for Tensor Dictionary Learning and 43.97 dB for BM4D. We regenerate that noise protocol so the comparison is direct. Any learned method that falls below 50.38 dB there has failed to absorb what the mixture model already knows.
For efficiency, the strongest competitor is score-based Top-K key pruning, as in Which Tokens to Use? and FastV from An Image is Worth 1/2 Tokens After Layer 2. We apply it to the same backbone at exactly our retained-key fraction, so both methods have matched FLOPs.
Three ablations isolate the key ingredient. The first replaces the mixture with a single heteroscedastic Gaussian variance head. The second uses a free learned sigmoid gate on keys with no likelihood term. The third drops the log-variance term from the equation. All variants get the same tuning budget, a fixed number of learning-rate and threshold trials.
Why not just…?
Why not just train SERT or SST longer and let attention learn to ignore corruption? Nothing in a mean-squared-error objective tells windowed attention that a stripe is unreliable rather than informative. SERT itself reports that purely learned models leave residual stripes. The stress test below, in which stripe and dead-line density increases, measures directly whether the gap between our method and the plain backbone grows with corruption. If it does not grow, this alternative wins and we say so.
Why not just predict a single per-pixel variance instead of a mixture? This is the most serious threat. A single Gaussian variance head is simpler and might capture most of the gain. The mixture should matter when impulse and stripe noise coexist with Gaussian noise, because one variance cannot represent "mostly fine, occasionally catastrophic". The mixture-versus-single-variance ablation settles this separately on Gaussian-only, impulse-only, and mixed noise. If the single variance matches the mixture on mixed noise, we will report the simpler model as the method.
Why not just run NMoG-LRMF first and feed its output to a fast network? Offline variational inference is slow; MoG-RPCA inverts rank-sized matrices at every iteration. Preprocessing also throws away the per-entry uncertainty the downstream network could use. We run this pipeline as a baseline and compare accuracy and wall-clock time.
Why not just prune keys by attention score? Score-based pruning keeps whatever looks salient, and structured corruption looks salient. The matched-FLOP comparison, together with a count of how many ground-truth corrupted pixels each method keeps as keys, tests this directly.
Proposed Experiments
We train on ICVL with the standard Gaussian, impulse, stripe, dead-line, and mixture protocols used by SST and SERT. We evaluate on ICVL test cubes and on Washington DC Mall under NMoG-LRMF's non-i.i.d. mixture noise. Real-noise tests use SERT's Realistic dataset, where SERT reports a 0.5 dB gain over MAC-Net, and the Urban dataset, for which no reported result was found on Lacuna. Metrics are PSNR, SSIM, and spectral angle. Cost is measured as GFLOPs, retained-key fraction, and runtime.
Three checks decide whether the idea holds. First, on synthetic noise, the per-key variance should separate corrupted from clean entries against the known masks, which we measure as area under the ROC curve. Second, at matched FLOPs, noise-aware key dropping should beat score-based Top-K, and its advantage should widen as stripe density rises. Third, the mixture should outperform the single-variance ablation on mixed noise but not on pure Gaussian noise. If the mixture gains nothing even on mixed noise, the mixture claim fails and only the simpler reliability-weighting claim survives.
References
- Transformer with a Mixture of Gaussian Keys
- Robust Principal Component Analysis with Complex Noise
- Denoising Hyperspectral Image With Non-i.i.d. Noise Structure
- Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising
- Spatial-Spectral Transformer for Hyperspectral Image Denoising
- Degradation-Noise-Aware Deep Unfolding Transformer for Hyperspectral Image Denoising
- Training Noise Token Pruning
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models