Scan the Abundance Maps, Not the Bands: Subspace State-Space Restoration for Hyperspectral Images

A hyperspectral image records hundreds of narrow color bands at every pixel. Most of those bands are redundant: a scene contains only a handful of distinct materials, so each pixel's spectrum is close to a mix of perhaps eight to sixteen basis spectra. Classical denoisers exploit this. They split the image into a small set of basis spectra and a few "abundance maps" saying how much of each basis is present at each pixel, then clean only the maps. The recent wave of hyperspectral state-space models takes a different route. These are recurrent sequence models that read an image pixel by pixel in linear time. They ignore the redundancy, run on the full band stack, and compensate with eight to twelve scan directions. We propose putting the two together in a specific way. The network learns a band-robust spectral basis inside the model, projects the noisy cube onto it, and runs a single-direction linear-time scan with a small local convolution over only the abundance maps. Basis and maps are refined alternately over a few unrolled stages.

The claim worth testing has two parts. The first is that the many-direction scans used in hyperspectral models such as SSUMamba and HSIDMamba are largely compensating for spectral entanglement. Once the bands are factored out, the cheap recipe that Adventurer found sufficient for ordinary images should be enough: one scan direction, reversed between layers, plus a prepended global-average token. The second is that the spectral basis must be estimated with per-band reliability weights, learned jointly with the spatial model. Otherwise a few dead or striped bands contaminate every abundance map, which is the "noise smearing" failure that NS3R documented for plain principal component analysis (PCA). If both hold, restoration cost becomes almost independent of band count. As a side effect, one trained network applies to sensors with different numbers of bands.

Where this sits

Two bodies of work motivate the proposal, and neither covers it. The classical subspace line factorizes the matricized cube $Y \in \mathbb{R}^{B\times N}$ ($B$ bands, $N$ pixels) as $Y \approx EZ$. Here $E \in \mathbb{R}^{B\times k}$ holds $k \ll B$ basis spectra and $Z \in \mathbb{R}^{k\times N}$ holds the abundance maps.

  • NGmeet alternates between estimating $E$ and running non-local nuclear-norm patch denoising on the maps. This makes runtime nearly independent of band count (about 48 s on the 192-band Washington DC Mall cube versus over 20 minutes for a full-band patch method).
  • NS3R replaces dense PCA with an elastic-net sparse PCA that zeroes corrupted-band loadings.
  • SNSSLrL regularizes $E$ for spectral smoothness and plugs a block-matching filter into an alternating optimization loop.

These methods supply our factorization and our concern about corrupted bands. Their spatial step, however, is hand-crafted patch matching, and their alternation is a fixed optimization loop rather than a learned one. ARGS-Diff is the closest deep relative. It runs a diffusion model on an eight-dimensional subspace for image fusion, with 21.85M parameters and 12 s per cube. Its basis is fixed, though, and its spatial network is a local convolutional U-Net, so distant pixels interact only through downsampling, which Guanbin Li et al. show blurs fine boundaries.

The deep linear-time line is the second body of work.

  • SSUMamba uses 12 directional paths over the raw cube plus 3D convolutions.
  • HSIDMamba uses 8 directions with spectral re-weighting.
  • HSRMamba reorders tokens by spectral similarity.
  • Dual Hyperspectral Mamba mixes global and windowed scans for compressive reconstruction.

All of them run on the full band dimension, so whenever a scan crosses bands, the sequence length grows with $B$. None decouples spectra from space. Adventurer constrains our design in a useful way. On ordinary images, multi-way scans buy little over a single direction with inter-layer flipping and a heading average token, which is 3.8× faster than Vim at 84.3% ImageNet top-1. No one has tested whether that finding survives in multi-band restoration. Our hypothesis is that it does, but only after spectral factorization.

Core Claim and Mechanism

Each of $T \approx 3$–$4$ unrolled stages does three things.

First, it estimates per-band reliability weights $w \in [0,1]^B$ from cheap statistics of the current residual: per-band noise variance, stripe energy, and saturation. A small learned network maps these statistics to weights. It then forms the basis $E$ from a truncated, differentiable singular value decomposition of the reliability-weighted current estimate.

Second, it projects the noisy data by weighted least squares rather than plain transposition:

$$Z = (E^\top W E)^{-1} E^\top W\, Y, \qquad W = \mathrm{diag}(w),$$

so that a band with $w_b \to 0$ contributes nothing to any abundance map. This is a learned, soft analogue of NS3R's zeroed loadings.

Third, it cleans the $k$ maps with a spatial engine. Each block is a uni-directional linear state-space scan (Mamba-2), with a global-average token prepended and the scan direction flipped between layers. A depthwise $3\times 3$ gated feed-forward layer follows each scan, in the spirit of ConvFormer and VmambaIR, to restore edges that long scans attenuate. The stage reconstructs $X = E\hat Z$ and passes it to the next stage.

Stripes and dead lines are not low-rank. A sparse residual branch, a few convolutions trained with an $L_1$ penalty following the low-rank-plus-sparse model of SNSSLrL, absorbs them before re-projection.

The cost per stage is $O(BkN)$ for projection and reconstruction plus $O(N d^2)$ for the spatial engine at width $d$. Only the projection depends on $B$, and it is a pair of matrix multiplications. Because the spatial engine never sees $B$, a network trained on 31-band data can in principle run on a 191-band cube. That is a concrete, testable consequence that no full-band model offers.

Research Program

The program has three parts.

The first part builds the model and checks the scan-count hypothesis directly. We sweep the number of scan directions (1 with flipping, 2, 4, 8) in two settings: on the abundance maps, and on the raw cube with the same block. We predict a widening gap. On the maps, one direction should match eight within 0.1–0.2 dB. On the raw cube, removing directions should cost noticeably more. If one direction suffices in both settings, the factorization is not what makes scans cheap, and the second claim falls back to a pure efficiency argument.

The second part isolates the reliability-weighted basis. We run a controlled band-corruption sweep: zero out, stripe, or saturate 0, 5, 10, and 20% of bands, then measure error on the uncorrupted bands. This shows whether corruption leaks across bands through the basis, the failure NS3R identified.

The third part studies rank and transfer. We sweep $k$ from 4 to 32 to map the trade-off that rank estimators currently set by hand. We also train on ICVL and apply the model without retraining to Washington DC Mall, Pavia, and Urban.

Baselines

The naive reference point is a two-stage pipeline. Plain PCA computes the basis from the noisy cube once, a strong 2D restoration network cleans the maps, and the result is projected back. We use VmambaIR's six-direction block for the 2D network so that the spatial engine is held roughly constant. This pipeline measures what end-to-end learning and band weighting add beyond "PCA plus a good network." We train it on the same data, give it the same parameter budget and the same hyperparameter search (an equal number of trials), and evaluate it on the same noise protocols. Beating it means a clear PSNR margin under mixed noise with dead bands. Near-parity under Gaussian noise is expected and acceptable.

The strongest existing deep competitor is SSUMamba, which reports 43.07 dB PSNR on ICVL mixture noise versus 40.54 dB for NSSNN (SSUMamba) with about 10M parameters. HSIDMamba reports 39.65 dB on ICVL at Gaussian $\sigma=70$ (HSIDMamba). We retrain both from released configurations on our exact splits and noise generators, with matched training iterations and GPU, rather than copying numbers. On ICVL, which has only 31 bands and therefore little room for spectral savings, success means matching SSUMamba within about 0.3 dB at lower latency and memory. On 191–210-band data, success means equal or better PSNR and spectral angle at a runtime that stays roughly flat as bands are added, while SSUMamba's grows.

The strongest training-free competitors are NS3R and NGmeet. NS3R reports 42.39 dB versus NGmeet's 41.98 dB on Pavia Centre at $\sigma=0.05$, with NS3R taking about 20 s (NS3R). SNSSLrL reports 31.77 dB on Pavia University mixed-noise Case 4 and a Q-metric of 0.0998 on real Urban data in 205 s (SNSSLrL). We run NS3R from its public code and NGmeet with the authors' default rank estimate. Beating them means higher PSNR and SSIM on synthetic Pavia and Washington DC Mall cases, plus no visible regression on the real Urban and Indian Pines cubes, at a small fraction of their runtime.

Two ablations isolate the key ingredients. In the first, the spatial engine on the maps is replaced by a parameter-matched convolutional U-Net, as in ARGS-Diff; this tests whether global linear-time context matters at all inside the subspace. In the second, the learned weights $w$ are set to all ones; this tests the reliability weighting.

Why not just…?

Why not just run PCA first and feed the maps to an existing restoration network? This is the most serious threat, and it may well match us under Gaussian noise, where PCA is near-optimal. The evidence against it is specific to corrupted bands. NS3R shows that dense PCA spreads water-absorption and dead-band noise into every coefficient image. NGmeet's gains come partly from re-estimating the basis after denoising, which a one-shot pipeline cannot do. The corruption sweep settles this. If the naive pipeline's error on clean bands stays flat as corrupted bands increase, the joint, weighted design is unnecessary, and we would report the result as an efficiency study of subspace scanning.

Why not just keep SSUMamba or HSIDMamba and accept the cost? On 31-band ICVL the cost is tolerable, and we do not expect to beat SSUMamba by much there. The argument is about high band counts. Their scans that cross bands lengthen with $B$, and their weights are tied to the training band count. We test this by subsampling Washington DC Mall to 32, 64, 128, and 191 bands and plotting latency, memory, and PSNR for each method. We also attempt cross-sensor transfer from ICVL, which those models cannot run without retraining.

Why not just use the classical subspace methods, which need no training? Their training-free generality is real, and on real data with unfamiliar noise they may beat any deep model trained on synthetic noise. The case for learning rests on runtime (20–205 s per cube for NS3R and SNSSLrL, versus an expected few seconds for a feed-forward pass) and on learned spatial priors. The real Urban and Indian Pines evaluation is where this could fail. We will report it regardless of outcome. If NS3R remains visibly better on real data, the honest conclusion is that our model is a fast approximation, not a replacement.

Why not just use a convolutional network on the maps, since convolutions are already linear time? In the subspace, the maps are only $k$ channels, so a U-Net is cheap. Its receptive field, however, grows only through depth and downsampling (Guanbin Li et al.). Stripes and dead lines span the whole image, which favors global context. The parameter-matched U-Net ablation answers this directly. A gap under stripe noise but not under Gaussian noise would confirm where global scanning pays off.

Decisive Checks

The work runs on ICVL for training and in-domain testing, Washington DC Mall and Pavia Centre and Pavia University for simulated remote-sensing noise, and HYDICE Urban and AVIRIS Indian Pines for real noise, using the case definitions of the cited papers. We report PSNR, SSIM, spectral angle, wall-clock time, and peak memory.

Three outcomes decide the proposal. The scan-direction sweep must show that one flipped direction suffices on the abundance maps but not on the raw cube. The corruption sweep must show that learned band weights keep clean-band error flat where plain PCA degrades. Runtime must stay near-constant as band count rises from 32 to 191 while matching SSUMamba-level PSNR. If the first fails, the paper becomes an efficiency result. If the second fails, the "PCA plus network" baseline wins, and we would say so.