A hyperspectral pixel records reflected light in hundreds of narrow wavelength bands. It is usually a blend of several materials, such as tree, water, soil and road. Spectral unmixing tries to recover two things from that blend: the pure spectrum of each material and the fraction of each material in each pixel. When light bounces between materials before reaching the sensor, the blend stops being a weighted average. The standard fix is to add a nonlinear correction term to the model. That makes the problem worse posed: a flexible correction term can explain the observed spectrum with the wrong fractions. Current deep unmixing models are small autoencoders trained from scratch on a single 100×100-pixel scene. Each one has to resolve this ambiguity using only the scene it is given.
We propose to resolve part of that ambiguity with knowledge learned elsewhere. A hyperspectral foundation model, pretrained without labels on hundreds of thousands of satellite patches, is kept frozen. Its features are fed only into the part of the unmixer that estimates fractions. The part that rebuilds the spectrum stays a physical mixing model: material spectra, a constrained nonlinear term, and fractions that are non-negative and sum to one. Many scenes are then unmixed jointly as separate tasks. Each scene keeps its own materials, and all scenes share the feature-based fraction estimator. The bet is that pretrained context helps most exactly where nonlinearity makes single-scene unmixing ill-posed. The experiments are built so that this bet can fail cleanly.
Core Claim
Frozen features from a hyperspectral foundation model reduce abundance error in nonlinear blind unmixing, measured against a parameter-matched spatial encoder trained from scratch. (An abundance is the fraction of a material in a pixel.) The claim has three conditions. The features must enter the encoder only. The decoder must still reconstruct the raw spectrum through a physical model. The nonlinear term must be kept from absorbing linear signal. If the claim holds, the gain should grow with the strength of the nonlinear mixing, because that is where the scene alone underdetermines the solution. Sharing one encoder across scenes, treated as a multi-task problem, should also help small or noisy scenes more than large clean ones.
What Exists and What Is Missing
Three unmixing papers supply the method and define the gap. Model-Based Deep Autoencoder Networks for Nonlinear Hyperspectral Unmixing (MAC-U) builds a decoder with a linear endmember branch plus a neural nonlinear branch. ("Endmember" is the term for a pure material spectrum.) Its encoder blends a pseudo-inverse projection with a learned correction. MAC-U is pixel-wise and single-scene, and we reuse its encoder–decoder coupling directly. Convolutional Autoencoder for Blind Hyperspectral Image Unmixing adds 3×3 spatial context in the encoder but keeps a purely linear decoder. Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders shows that building the non-negativity and sum-to-one constraints into the bottleneck activations works well, again with a linear decoder and per-dataset training. None of the three uses pretrained representations.
The closest work is Solving Spectrum Unmixing as a Multi-Task Bayesian Inverse Problem with Latent Factors for Endmember Variability (BIPU). It is a variational autoencoder with separate latent factors for abundance and for spectral variability, plus an auxiliary head that predicts which materials are present. BIPU is multi-task only within one dataset, uses feedforward encoders trained from scratch, and models nonlinearity as additive spectral shifts rather than inter-material scattering. Our proposal differs on all three axes. The new knowledge comes from frozen pretrained features, the multi-task structure runs across scenes, and the decoder carries an explicit scattering-style nonlinear term.
On the pretraining side, SpectralEarth: Training Hyperspectral Foundation Models at Scale pretrains backbones on 538,974 EnMAP patches with 202 bands. It places a 1D convolutional spectral adapter in front of standard vision backbones and reports transfer to the DESIS and Hyperion sensors. It evaluates only classification, segmentation and biomass regression, never unmixing. SpectralGPT and SpectralX likewise target perceptual tasks. SpectralGPT works on 12-band Sentinel-2 data, which is too coarse for material spectra, so we treat it as a constraint and not as a candidate backbone.
One finding cuts against us. Does the Data Processing Inequality Reflect Practice? frames the risk that any fixed processing step discards information. A backbone pretrained for masked reconstruction or contrastive objectives may smooth away exactly the low-amplitude spectral detail that separates materials. This constraint shapes the design: the raw spectrum always reaches the encoder through the pseudo-inverse branch, and it is always the reconstruction target.
Research Program
For scene $s$ with pixel spectrum $y \in \mathbb{R}^B$, endmember matrix $M_s \in \mathbb{R}^{B \times P_s}$ and abundances $a$, the decoder is
$$\hat{y} = M_s a + \psi_s(M_s, a),$$
where $\psi_s$ is a small network whose output is penalized in magnitude. Its input includes bilinear products $a_i a_j (m_i \odot m_j)$ of pairs of material spectra, so its default form is the physical second-order scattering term. The encoder is
$$a = \sigma\big(\alpha\, M_s^{\dagger} y + (1-\alpha)\, h_s(\,g(F(Y_{\text{patch}})))\big).$$
In this expression, $M_s^{\dagger}$ is the pseudo-inverse of the endmember matrix. $F$ is the frozen SpectralEarth backbone applied to the image patch $Y_{\text{patch}}$ around the pixel. $g$ is a trainable projector shared across all scenes. $h_s$ is a small scene-specific head that maps shared features to that scene's $P_s$ materials. $\sigma$ is a softmax that enforces non-negativity and sum-to-one, and $\alpha$ is learned as in MAC-U. Training minimizes the sum of per-scene reconstruction losses plus $\beta\lVert\psi_s\rVert^2$.
Training on several scenes at once is where task balancing enters. Scenes differ in size, noise and band count, so their gradients on the shared projector $g$ can conflict. We compare two ways of handling this. The first is equal weighting. The second is RotoGrad, which rescales task gradient magnitudes and rotates the shared representation per task.
Sensor bands have to be made compatible first. Jasper Ridge, Samson and Urban have 198, 156 and 162 bands from AVIRIS/HYDICE-type sensors. These are resampled to EnMAP's 202-band grid by spectral-response interpolation before $F$ is applied, while the decoder reconstructs each scene's native bands. SpectralEarth's reported cross-sensor transfer makes this plausible, but the resampling is itself a risk, and the diagnostic below tests it first.
The work proceeds in three stages. The first is a diagnostic: on scenes with known abundances, can a ridge probe on frozen features predict abundances better than the same probe on raw spectra plus a 3×3 neighbourhood? If not, the project stops or pivots to fine-tuning. The second stage builds semi-synthetic scenes where nonlinearity strength is a dial. We take USGS library spectra resampled to EnMAP bands, mix them with bilinear and post-nonlinear models in the style of MAC-U's DC1/DC2 cubes, and lay the mixtures out using abundance maps derived from real EnMAP-CORINE and EnMAP-CDL label rasters. The result has realistic spatial structure and exact ground truth. Real scenes of this kind, rather than random fields alone, give the foundation features a fair chance to carry useful context. The third stage moves to real benchmarks and to held-out-scene amortized unmixing, where the shared projector is frozen and only $h_s$, $M_s$ and $\psi_s$ are fitted on a new scene.
Baselines
The naive reference is fully constrained least squares with endmembers from Vertex Component Analysis, together with the plain linear-decoder autoencoder of the Raman study. These are the floors that any nonlinear method must clear on nonlinear data, and both run per scene with their standard settings.
The strongest existing competitor is BIPU. It is the only cited method that already combines autoencoder unmixing with an auxiliary task, and it reports an aRMSE of 7.02 on Jasper Ridge against 7.54 for the next-best baseline (BIPU). It also reports an 18.6%–31.48% aRMSE reduction over eight baselines on FENIR. We will quote those numbers but also rerun BIPU under our own protocol, with the same reference abundances, the same endmember-matching rule and the same random seeds. Beating it means lower abundance RMSE on Jasper Ridge under that shared protocol. On the semi-synthetic suite, the margin over BIPU should exceed seed-to-seed spread at moderate and strong nonlinearity.
MAC-U is the strongest nonlinear-decoder competitor. No reported result was found on Lacuna for MAC-U as a number, because its DC1/DC2 and Urban results are given only qualitatively in the file. We reimplement it and run it on our semi-synthetic cubes and on Urban. For CAE on Jasper Ridge and Samson, no reported result was found on Lacuna for CAE as a numeric RMSE.
The ablation that isolates the key ingredient keeps our full architecture and replaces the frozen backbone $F$ with a spatial-spectral CNN trained from scratch. That CNN gets the same receptive field, the same number of trainable parameters as $g$, and the same tuning budget of 30 random-search trials per method on a held-out validation scene. A second ablation removes the cross-scene sharing by training one model per scene. A third swaps RotoGrad for equal weighting. All methods use identical data splits, endmember counts, and wall-clock budgets per scene, excluding the one-time feature extraction for $F$, which we report separately.
Why not just…?
Why not just add spatial context with a convolutional encoder, as CAE does, and skip the foundation model? This is the most serious threat. Much of the gain from pretrained features might simply come from looking at neighbouring pixels. The from-scratch spatial-encoder ablation exists to settle it. If that ablation matches the frozen-feature model at every nonlinearity level, the contribution of pretraining is nil and we will report it that way. The prediction that separates the two is a growing gap with nonlinearity strength and with fewer pixels per scene, since a from-scratch encoder can only learn priors from the scene in front of it.
Why not just penalize the nonlinear term harder to fix identifiability? A strong penalty on $\psi$ does stop it from swallowing linear signal. It does so by biasing the model back toward linear mixing, which is wrong exactly where nonlinearity is real. We sweep $\beta$ for every method, including the baselines. We also report a leakage measure on semi-synthetic data: the energy explained by the nonlinear branch compared with the true nonlinear energy. Our claim requires lower abundance error at matched leakage, not just a different point on the same trade-off curve.
Why not just fine-tune the foundation model, or fit a supervised probe from its features to abundances? Blind unmixing has no abundance labels on real scenes, so a supervised probe is available only as a diagnostic. We use it that way, in the first-stage check. Full fine-tuning against a reconstruction loss on a 10,000-pixel scene risks the overfitting that frozen use avoids. Still, SpectralEarth reports that tuning its spectral adapter alone approaches full fine-tuning. We therefore include an adapter-only tuning variant. If it beats the frozen version, we adopt it.
Why not just pool all scenes and train one single-task model? Pooling with equal weighting is exactly the multi-task arm without balancing, so the comparison is built in. If equal weighting matches RotoGrad, the balancing machinery is dropped. The claim about sharing is then tested only by per-scene versus shared training.
Proposed Experiments and Decisive Checks
The first decisive check is the linear-probe diagnostic on semi-synthetic scenes. Frozen features must predict abundances with lower RMSE than raw-spectrum-plus-neighbourhood features, or the program changes course before any unmixing model is built.
The second check sweeps nonlinearity strength on the semi-synthetic bilinear and post-nonlinear suites. We report abundance RMSE, endmember spectral angle and nonlinear leakage, averaged over five seeds, for our model, MAC-U, BIPU, the linear floors and the from-scratch-encoder ablation. The core claim survives only if our model's advantage over the matched from-scratch encoder is positive and increases with nonlinearity.
The third check uses the real scenes Jasper Ridge, Samson and Urban, with abundance RMSE against their standard reference maps, and compares against the rerun BIPU and its reported 7.02 on Jasper Ridge.
The fourth check is held-out-scene amortization. We train the shared projector on all but one scene, fit only the scene-specific parts on the held-out scene, and compare against per-scene training at 10%, 25% and 100% of that scene's pixels. Sharing should matter most at the smallest fraction.
A negative result in the second check is informative in its own right. It would show that current hyperspectral foundation features, trained for perceptual tasks, do not encode the material-fraction structure that physical inversion needs. It would also locate where that information is lost, by comparing the probe results for SpectralEarth checkpoints pretrained with masked reconstruction against those pretrained with contrastive objectives.
References
- Model-Based Deep Autoencoder Networks for Nonlinear Hyperspectral Unmixing
- Convolutional Autoencoder for Blind Hyperspectral Image Unmixing
- Hyperspectral Unmixing for Raman Spectroscopy via Physics-Constrained Autoencoders
- Solving Spectrum Unmixing as a Multi-Task Bayesian Inverse Problem with Latent Factors for Endmember Variability
- SpectralEarth: Training Hyperspectral Foundation Models at Scale
- SpectralGPT: Spectral Remote Sensing Foundation Model
- SpectralX: Parameter-efficient Domain Generalization for Spectral Remote Sensing Foundation Models
- Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks
- RotoGrad: Gradient Homogenization in Multi-Task Learning