Open-vocabulary segmentation labels every pixel of an image with whatever category a user types, such as "greenhouse roof" or "flooded field," by matching image regions against text embeddings from a vision-language model like CLIP. Until now it has been built and tested only on clean three-band color photographs. Hyperspectral sensors record hundreds of narrow wavelength bands per pixel. Their raw data is rarely clean: individual bands carry different amounts of random noise, and detector faults paint stripes, dead lines and isolated bright pixels across the image. The usual way to handle this is to fully restore the cube with an expensive denoiser and then hand a color rendering to the segmenter. We argue this does the wrong job. A frozen vision-language model only ever sees three rendered channels, so most of the restoration effort is wasted. Meanwhile the one thing a denoiser computes that the segmenter actually needs, namely where the data cannot be trusted, is thrown away.
The proposal in two sentences: split the noisy cube cheaply into a smooth low-rank part, a sparse corruption part, and band-wise noise levels. Then use the corruption estimates to decide which bands are rendered, which image patches the frozen vision-language and segment-anything models are allowed to attend to, and how much each patch's text-match score counts, instead of restoring the cube to high fidelity first. We expect this "gate, don't fully restore" route to match or beat denoise-then-segment pipelines on unseen categories at a fraction of the runtime. We also expect it to show that low-level restoration scores such as peak signal-to-noise ratio (PSNR) are a poor predictor of downstream open-vocabulary accuracy.
Core Claim
The standard hyperspectral noise model writes the observation as $Y = X + S + N$. Here $X$ is the clean, low-rank signal (neighbouring bands are highly correlated), $S$ is the sparse structured corruption (stripes, dead lines, impulses), and $N$ is dense band-dependent Gaussian noise. Restoration methods such as LRTDTV (Tucker decomposition plus spatial-spectral total variation) and FastHyMix (mixture-model separation of Gaussian and sparse residuals with subspace learning and a plug-in deep denoiser) estimate $S$ and the per-band noise levels on the way to producing $\hat X$. They then report only $\hat X$.
Our claim is that $\hat S$ and the per-band noise variances $\hat\sigma_b^2$ are more valuable to a frozen open-vocabulary segmenter than the extra decibels of fidelity in $\hat X$. Concretely, we turn them into a per-patch reliability weight. For a patch $p$ of the rendered image,
$$ r_p = \exp\!\Big(-\lambda \cdot \tfrac{1}{|p|}\sum_{(i,j)\in p}\sum_{b\in\mathcal{R}} |\hat S_{ijb}|/\hat\sigma_b\Big), $$
where $\mathcal{R}$ is the set of bands used for rendering, $|p|$ is the number of pixels in the patch, and $\lambda$ is a single temperature. This weight is used in two ways. It masks low-reliability patches as keys in the frozen encoders' attention, so stripe tokens cannot contaminate their neighbours. It also down-weights those patches' image-text similarity scores before spatial aggregation, so their labels are filled in from reliable context.
Band selection uses $\hat\sigma_b$ directly. Within each of three spectral windows used for rendering, we pick the least-noisy band rather than a fixed red/green/blue index. A second, trainable stream carries the spectral information that three channels discard. It takes the low-rank subspace coefficients of $\hat X$ as input and is fused with frozen CLIP in the generalist-plus-specialist arrangement of GSNet.
What is new is not the combination of hyperspectral input with open-vocabulary segmentation per se. The research file found no paper doing either that combination or the noise-aware version. The new element is the specific mechanism: noise-decomposition outputs act as attention and score gates for frozen vision-language models, replacing full restoration.
How This Relates to Prior Work
Three lines of work motivate the gap. Denoising Hyperspectral Image with Non-i.i.d. Noise Structure establishes that real hyperspectral noise varies from band to band. This is why a single global "denoise" step is a blunt instrument and why per-band estimates are informative. Degradation-Aware Metric Prompting (DAMP) computes six cheap physical statistics from the degraded cube and uses them to route a restoration mixture-of-experts. This is the closest precedent for "measure the degradation and act on it." However, DAMP acts only to restore pixels, is scored by PSNR, SSIM and spectral angle, and has no statistic for directional stripes or dead lines. We instead act on the measurement inside a segmenter and target structured corruption explicitly. MoGLRMF with MRF post-processing is the only paper in the file that couples mixed-noise modelling with semantic labelling. Its labels come from closed-set classifiers over a handful of fixed classes, and we keep only its lesson that noise-robust low-rank features help labelling.
Two open-vocabulary remote-sensing systems supply the segmentation machinery and also constrain the design. AerOSeg uses a frozen segment-anything encoder for boundaries, four-way rotation of the CLIP input, and a back-projection regulariser against overfitting to seen classes. GSNet pairs frozen CLIP with a DINO-pretrained remote-sensing stream. Its finding that swapping in a remote-sensing-tuned CLIP hurts open-vocabulary accuracy is a negative result that rules out end-to-end retuning of the language-aligned encoder, so our spectral stream sits beside frozen CLIP, never inside it. Both systems assume pristine RGB. LoGoSeg shows that open-vocabulary accuracy hinges on local region alignment and on suppressing hallucinated categories. That sensitivity to local evidence is exactly why corrupted local patches matter, and its object-existence prior is a natural place to add the reliability weight.
Research Program
The work has three parts.
The first part is a measurement study. Nothing in the file reports how open-vocabulary segmenters degrade under sensor-style corruption, so we inject synthetic stripes, dead lines, impulse noise and band-dependent Gaussian noise into the test images of the RGB remote-sensing benchmarks. We follow the four noise cases used by FastHyMix (Gaussian, Gaussian plus stripes, Gaussian plus salt-and-pepper, all three combined) and add dead lines. We then measure how far GSNet and AerOSeg fall. This tests the file's conjectured failure, that segment-anything boundaries fragment along stripes and outlier tokens corrupt CLIP's similarity maps. It also tells us whether the problem is large enough to matter before we build anything.
The second part builds the gated pipeline. It has four components. The first is a fast decomposition: a randomized-SVD low-rank fit with an $\ell_1$ sparse term, run for a fixed small number of iterations, with per-band noise estimated from residuals as in FastHyMix. The second is noise-aware band selection for rendering. The third is reliability-masked attention and score weighting inside frozen CLIP and segment-anything. These three components are training-free. The fourth is a lightweight spectral stream on subspace coefficients, trained only on base classes and fused with frozen CLIP via GSNet-style query-guided fusion.
The third part is an analysis of which restoration properties actually predict open-vocabulary accuracy. For many denoisers and settings, we plot PSNR, spectral angle and residual-stripe energy against unseen-class intersection-over-union. The question is whether fidelity metrics, structured-residual metrics, or neither track the downstream task.
Baselines
The naive reference point runs AerOSeg and GSNet directly on a fixed true-color rendering of the noisy cube, with no denoising and no gating. This sets the floor and, compared with the clean-cube rendering, measures the size of the noise penalty.
The strongest competitor is the sequential pipeline: restore with a strong mixed-noise method, render, then segment. We run it with three restorers that span the design space. FastHyMix is fast and parameter-free. LRTDTV is the classical tensor-plus-total-variation model; it reports 38.63 dB mean PSNR against 36.35 dB for LRTV on simulated HYDICE Urban mixed noise. DAMP is the learned option, using the public MiliLab/DAMP weights. This is the strongest competitor because it is exactly what a practitioner would do today, and the best of these restorers removes most stripes before the segmenter sees them.
For fairness, every pipeline uses the same segmenter checkpoints, the same rendering windows, the same seen/unseen splits, and an equal hyperparameter search budget. For the restorers, that budget covers their documented knobs, such as subspace dimension; for our method, it covers $\lambda$ and the number of decomposition iterations. Wall-clock time is measured on the same GPU and CPU.
No reported result was found on Lacuna for open-vocabulary segmentation on hyperspectral data, clean or noisy. The nearest reported anchors are on RGB. GSNet reports 31.25% average mIoU across its four test datasets when trained on LandDiscover50K, and 45.75% mIoU on ISPRS Potsdam. AerOSeg reports a 2.54 h-mIoU average gain over the prior OVRS model on iSAID, DLRSD and OpenEarthMap. In the RGB corruption study, we will first reproduce these clean numbers. "Beating" the strongest competitor on hyperspectral data means higher unseen-class and harmonic-mean IoU than the best restore-then-segment pipeline at equal or lower runtime. Equal accuracy at roughly an order of magnitude less time also counts as a meaningful win, given that FastHyMix itself reports 7 seconds against 203 seconds for KBR on one Tiangong-1 image.
Two further references matter. One is a closed-set spectral-spatial classifier with a confidence-threshold reject option, run through the SHIP package. The other is a CLIP whose first layer is widened to all bands and fine-tuned end to end under noise augmentation.
Ablations isolate the key ingredient. We compare reliability gating driven by $\hat S$ against three alternatives: a random mask of equal coverage, a mask from a naive per-band median-deviation outlier detector, and no mask. We also test noise-aware band selection alone, gating alone, and the spectral stream alone.
Why not just…?
Why not just denoise with FastHyMix or DAMP and feed the rendering to AerOSeg? This is the real threat, and the proposal is designed to settle it rather than argue it away. The file's evidence against it is indirect. FastHyMix notes that a wrong subspace dimension under heavy mixed noise leaves structured residuals. LRTDTV acknowledges that global low-rank and total-variation constraints blur fine detail. Optimisation-based restorers are slow. None of these show that the downstream segmenter actually suffers. Our restorer-sweep analysis does test this. If the best restore-then-segment pipeline matches our gated method on unseen classes at similar cost, we will report that the open-vocabulary problem on noisy hyperspectral data reduces to restoration plus band selection. The contribution then becomes the benchmark and the finding that PSNR does or does not predict segmentation accuracy.
Why not just drop the noisy bands and render from the clean ones? For a frozen three-channel model, this simple heuristic may capture much of the benefit, because noise-aware band selection is already one of our components. It cannot handle stripes and dead lines that hit the chosen bands at specific columns, and it gives no signal to the attention layers. The ablation of band selection alone versus band selection plus gating measures exactly how much the spatial reliability map adds. If it adds nothing, the method should be simplified to band selection.
Why not just widen CLIP's input to all bands and fine-tune it on noisy data? GSNet reports that replacing generalist CLIP with a remote-sensing-tuned version degrades open-vocabulary performance. AerOSeg and LoGoSeg each needed extra regularisers just to stop ordinary fine-tuning from collapsing onto seen classes. Retraining the language-aligned encoder on a few small hyperspectral scenes should therefore hurt unseen classes most. We include this model as a baseline and expect it to win on seen classes and lose on unseen ones. A lower harmonic mean would confirm the expectation.
Why not just use a closed-set classifier with an "unknown" threshold? It can flag novel pixels but cannot name or separate them, so it answers a different question. We report its unknown-detection rate under noise as a reference. The prediction, consistent with MoGLRMF's observation that noise degrades features, is that its confidence threshold becomes unreliable as corruption grows.
Proposed Experiments
The first experiment is the RGB corruption sweep on the GSNet test sets (FLAIR, FAST, ISPRS Potsdam, FloodNet, after LandDiscover50K training) and the AerOSeg benchmarks (iSAID, DLRSD, OpenEarthMap). We first reproduce the clean numbers, then report mIoU and h-mIoU as stripe density, dead-line count and impulse rate increase, with and without our training-free gating. The rendering has three channels here, so this isolates the gating mechanism from any spectral benefit.
The second experiment is the hyperspectral benchmark. We build cross-scene seen/unseen splits from the labelled scenes in the file (Indian Pines, Pavia University, Pavia Center, Kennedy Space Center, and, where labels are available, Xiong'an, Chikusei and HyRank). Each split trains on base classes from some scenes and tests on held-out class names elsewhere, under the five synthetic noise cases. For Indian Pines, we additionally evaluate on the full 220-band cube, whose noisier bands supply real rather than simulated corruption with ground-truth labels. Metrics are seen, unseen and harmonic-mean IoU, plus wall-clock time per scene. The restorers are also scored with PSNR, SSIM and spectral angle, which feed the correlation analysis.
The third experiment is a qualitative check on real, unlabeled noisy data (HYDICE Urban, Hyperion Cuprite, Tiangong-1). It examines whether segment-anything masks follow stripes or object contours, and whether the reliability map lines up with the visible defects.
The decisive check is the comparison between our method and the best restore-then-segment pipeline on unseen-class IoU on noisy hyperspectral splits, read together with the gating-versus-random-mask ablation. Our claim holds only if structure-derived gating beats both the random mask and band selection alone, and matches or beats full restoration at lower cost.
The main risk is honest and specific. These hyperspectral scenes are small, coarse-resolution and have few classes, so frozen CLIP may perform poorly even on clean renderings. In that case all methods sit near the floor and comparisons are uninformative. We will report the clean-cube ceiling first. If it is too low, we will restrict the claims to the RGB corruption study, where the anchoring numbers exist.
References
- Hyperspectral Image Restoration Via Total Variation Regularized Low-Rank Tensor Decomposition
- FastHyMix: Fast and Parameter-Free Hyperspectral Image Mixed Noise Removal
- Towards Open-Vocabulary Remote Sensing Image Semantic Segmentation
- Denoising Hyperspectral Image With Non-i.i.d. Noise Structure
- Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
- Spectral-Spatial Hyperspectral Image Classification via Robust Low-Rank Feature Extraction and Markov Random Field
- AerOSeg: Harnessing SAM for Open-Vocabulary Segmentation in Remote Sensing Images
- LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation