A hyperspectral camera records hundreds of wavelengths per pixel. Each pixel usually covers several materials at once, such as soil, water and leaves. Unmixing recovers which pure materials (called endmembers) are present and in what fractions (called abundances). The standard assumption is that a pixel's spectrum is a weighted average of the pure spectra. That assumption breaks when light bounces between materials before reaching the sensor, for example inside a tree canopy or among grains of a bright mineral powder. Recent deep unmixers add a second, free-form "nonlinear" branch that fits whatever the average cannot explain.
This proposal starts from a failure these models invite. Because the nonlinear branch can explain anything, it also absorbs errors in pixels where mixing really is linear, and the abundance estimates degrade there. We propose to tell the network, pixel by pixel, how much multiple scattering to expect. The signal comes from cheap, deterministic physical measurements taken from the image: an estimated brightness (albedo), how poorly a linear fit explains the pixel, and how curved its spectrum is. These measurements act as a dynamic prompt, meaning an input-dependent conditioning signal. The prompt bounds and steers the nonlinear branch. A linear-time spatial sequence model then lets neighbouring pixels share evidence about which scattering regime they belong to.
The second contribution is the benchmark. Today, nonlinear unmixing is scored mostly on cubes built with a simple bilinear formula, or on real scenes whose "ground truth" varies with preprocessing choices. We will build and release a physically simulated benchmark in which nonlinearity varies across the scene in a controlled way. Without such a benchmark, the central claim cannot be tested.
Core Claim
In scenes where nonlinearity is spatially uneven, a two-stream unmixer whose nonlinear branch is gated by physically computed per-pixel prompts will recover abundances more accurately than an ungated two-stream unmixer, in both the linear and the strongly nonlinear parts of the scene. The ungated baseline should show a measurable penalty in low-albedo, near-linear regions, where its nonlinear branch "explains away" linear signal. The gated model should not.
Two alternatives could match the gain, and the ablations test both directly. The first is a learned, black-box gate that sees the same pixel. The second is a local convolution in place of the global scan. The idea is new only if physically defined prompts beat learned ones out of distribution, the setting where Degradation-Aware Metric Prompting for Hyperspectral Image Restoration found them most useful.
Grounding in Prior Work
Several papers motivate the problem.
- Structured Sparse Method for Hyperspectral Unmixing and Unmixing Hyperspectral Data show that ignoring multi-path scattering biases abundances in complex terrain.
- Domain studies on layered mural pigments (Research on the Hyperspectral-based Unmixing Model of Composite Pigments on Mural Surfaces) and brain tissue (Redefining Spectral Unmixing for In-Vivo Brain Tissue Analysis from Hyperspectral Imaging) find linear models inadequate. This proposal does not target those applications.
- Spectral Unmixing: A Derivation of the Extended Linear Mixing Model From the Hapke Model derives the Extended Linear Mixing Model (ELMM) as a first-order expansion of Hapke radiative transfer, a physical model of scattering in particulate media. It also shows the approximation fails for bright materials such as Tephra. This result shapes the design: it tells us that albedo is the physically meaningful axis along which nonlinearity should be gated, and it gives us a simulator.
The method borrows from three sources.
- The two-stream decoders of PhI-Net (A Deep Disentangled Approach for Interpretable Hyperspectral Unmixing) and ID-Net (Learning Interpretable Deep Disentangled Neural Networks for Hyperspectral Unmixing) pair a linear branch with a nonlinear residual branch and use Dirichlet priors to keep abundances non-negative and summing to one. We keep this structure. We change two things those models leave open: they apply no constraint on where the residual is allowed to be large, and they process each pixel in isolation. ID-Net's authors name spatial context and cost as future work.
- DAMP computes deterministic statistics from the input cube, such as a high-frequency energy ratio and mean spectral curvature, and routes features through experts. We reuse the idea and its code (MiliLab/DAMP). Unlike DAMP, our prompts describe scattering regime rather than sensor corruption, and they bound a physical decoder rather than a restoration network.
- CoCoOp (Conditional Prompt Learning for Vision-Language Models) shows that input-conditioned prompts generalize better than fixed ones. Its learned meta-network becomes our main ablation rather than our method.
Two spatial models constrain the design. The spatial carrier is a visual state space model, a recurrent scan over pixels whose cost grows linearly with image size. We adopt the eight-direction scan of RS-Mamba for Large Remote Sensing Image Dense Prediction. That paper also warns that very long scans pull clutter into the hidden state, so we scan only prompt-conditioned features and not raw spectra. Following FusionMamba: Efficient Remote Sensing Image Fusion With State Space Model, where one stream modulates the scan parameters of another, we let the prompt modulate the scan's step size. This lets regime boundaries, such as canopy edges, reset context.
Research Program
The model writes each pixel spectrum $\mathbf{y}$ as
$$\mathbf{y} = \psi\,\mathbf{M}\mathbf{a} + g(\mathbf{p})\,\mathbf{r}(\mathbf{a}, \mathbf{h}) + \mathbf{n},$$
where each term is as follows:
- $\mathbf{M}$ holds the endmember spectra.
- $\mathbf{a}$ is the abundance vector on the simplex.
- $\psi$ is the ELMM scaling factor.
- $\mathbf{r}$ is a nonlinear residual decoder that sees abundances and the spatial context $\mathbf{h}$ produced by the scan.
- $\mathbf{n}$ is noise.
- $g(\mathbf{p}) \in [0,1]$ is a per-pixel gate computed from the physical prompt $\mathbf{p}$.
The prompt has three components, all computed in closed form:
- A single-scattering-albedo proxy, obtained by inverting a simplified Hapke reflectance relation.
- The normalized residual of a fast fully constrained least squares (FCLS) fit.
- A spectral-curvature statistic, taken from DAMP.
The gate is a small monotone function of these components, so that brighter pixels and pixels with larger linear misfit are allowed larger residuals. Training is self-supervised on reconstruction. Following ID-Net, endmembers are initialized with vertex component analysis, and pretraining uses scene-tailored synthetic data.
The benchmark, called Hapke-Var, uses the Basalt, Palagonite and Tephra laboratory spectra from the Hapke–ELMM study plus library spectra. Abundance maps are drawn from spatially correlated random fields. Reflectance is generated with intimate Hapke mixing: materials are mixed in albedo space and then converted to reflectance. Incidence and emergence angles vary with synthetic terrain, and patches of macroscopic two-bounce mixing mimic canopies. Each cube comes with an exact abundance map and a per-pixel "true nonlinearity" map, defined as the distance of the Hapke reflectance from its ELMM approximation.
For real scenes, we fix one preprocessing protocol: the band subset, the reference abundances used, and the endmember initialization and its seeds. We publish it with all baseline runs. This addresses the variance across papers that the research file identifies as a reproducibility problem.
Baselines
The naive reference is FCLS under the linear model. PhI-Net's authors report it at 2.45 seconds on Samson. It sets the floor for both accuracy and speed.
The strongest linear competitor is EDAA (Entropic Descent Archetypal Analysis for Blind Hyperspectral Unmixing). It models endmembers as convex combinations of observed pixels and reports the lowest abundance RMSE on five of six real benchmarks (Samson, Jasper Ridge, Urban4, Urban6, APEX, WDC), with 75 seconds on Urban6. No per-dataset RMSE for Samson or Jasper Ridge was found on Lacuna, so we will rerun EDAA under our fixed protocol rather than quote a number. ELMM is run alongside it as the physics-based linear option.
The strongest nonlinear competitors are PhI-Net and ID-Net.
- PhI-Net reports a 20–31% NRMSE improvement over DeepGUn on DC1/DC2, at about 1,606 seconds on Samson.
- ID-Net reports a 39% NRMSE improvement on nonlinear synthetic data and 25% under spectral variability.
- No numerical Samson or Jasper Ridge RMSE for ID-Net was found on Lacuna.
All methods get the same cubes, the same endmember initialization, the same seeds and the same GPU. Deep baselines receive a matched hyperparameter search budget. Beating them means two things. On DC1/DC2 and Hapke-Var, we need lower abundance RMSE than the better of PhI-Net and ID-Net, with confidence intervals over seeds that do not overlap. On real scenes, we need no loss relative to EDAA while running at least an order of magnitude faster than PhI-Net.
Three ablations isolate the ingredients:
- Remove the gate ($g \equiv 1$), which reduces the model to a spatial PhI-Net.
- Replace the physical prompt with a CoCoOp-style learned meta-network given the raw spectrum, with parameters matched.
- Replace the eight-direction scan with a convolutional encoder of equal parameter count, and separately with no spatial module.
Why not just…?
Why not just use ELMM, since it already absorbs illumination and topography? ELMM is a first-order expansion around zero albedo, and its own derivation shows larger errors for bright Tephra. On Hapke-Var we will plot abundance error against true albedo. We expect ELMM to be competitive in dark bins and to degrade in bright ones. If it does not degrade, the case for any nonlinear branch weakens, and we will report that result.
Why not just take EDAA and add spatial smoothing? This is a serious threat on real scenes such as Samson, where much of the mixing may be close to linear. Smoothing cannot correct a systematic bias from multiple scattering in bright or canopy regions. It can only average that bias over neighbours. The decisive test is error within the high-nonlinearity bins of Hapke-Var. We will run EDAA with total-variation smoothing as an additional baseline. If it matches our model on real scenes, we will say so and limit the claim to strongly nonlinear regimes.
Why not just let the network learn its own gate? A learned gate sees the same pixel and could, in principle, discover albedo. Our argument is not that it cannot. It is that a learned gate will fit the nonlinearity pattern of the training distribution and fail when that pattern shifts, as DAMP observed for learned prompts under unseen degradations and CoCoOp observed for static prompts on unseen classes. The test trains on one Hapke-Var material set and geometry range and tests on held-out ones. If the learned gate matches the physical prompt there too, the physical prompt reduces to an interpretability feature, and we will present it that way.
Why not just use a CNN for spatial context? On 95×95 Samson, a CNN may well tie with the scan, and we expect this. The scan's advantage should appear only on large scenes and long-range structures, where RS-Mamba showed that Transformers run out of memory and patching fragments long objects. We will compare equal-parameter CNN and scan versions on full-size Urban, APEX and WDC and on large Hapke-Var tiles. If the CNN matches everywhere, we drop the state-space component and keep the gated decoder, which is the core contribution.
Proposed Experiments
The decisive experiment is the leakage test on Hapke-Var. For every pixel we record the true nonlinearity, the magnitude of the estimated residual $g\,\mathbf{r}$, and the abundance error. The claim holds only if two conditions are met:
- In near-linear pixels, the ungated two-stream model shows a large residual and a higher abundance error than FCLS, while our model does not.
- In strongly nonlinear pixels, our model matches or beats the ungated model.
The second experiment is generalization under held-out materials and geometries, comparing physical and learned gates. The third is scaling. We report wall-clock time and memory against image size, from 95×95 up to tiled strips of several thousand pixels on a side, against PhI-Net's roughly 1,606 seconds and EDAA's 75 seconds on Urban6.
On real scenes (Samson, Jasper Ridge, Urban4/6, APEX, WDC and Cuprite), we report:
- abundance RMSE under the fixed protocol,
- endmember spectral angle,
- variance across five seeds for every method.
We will treat these as consistency checks, because their reference abundances are not physical ground truth. The released benchmark, protocol and baseline runs are deliverables in their own right.
References
- Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
- Structured Sparse Method for Hyperspectral Unmixing
- Unmixing Hyperspectral Data
- Research on the Hyperspectral-based Unmixing Model of Composite Pigments on Mural Surfaces
- Redefining Spectral Unmixing for In-Vivo Brain Tissue Analysis from Hyperspectral Imaging
- Spectral Unmixing: A Derivation of the Extended Linear Mixing Model From the Hapke Model
- A Deep Disentangled Approach for Interpretable Hyperspectral Unmixing
- Learning Interpretable Deep Disentangled Neural Networks for Hyperspectral Unmixing
- Conditional Prompt Learning for Vision-Language Models
- RS-Mamba for Large Remote Sensing Image Dense Prediction
- FusionMamba: Efficient Remote Sensing Image Fusion With State Space Model
- Entropic Descent Archetypal Analysis for Blind Hyperspectral Unmixing