When a remote sensing satellite or airborne sensor captures an image, each pixel typically covers a macroscopic area on the ground. Consequently, the light collected within a single pixel is rarely reflected from a single, pure material. Instead, it is a mixture of spectral signatures from various constituent substances, known as endmembers (such as soil, water, or specific vegetation types).
To make sense of this data, researchers use spectral unmixing: the process of decomposing an observed pixel's spectrum into its constituent endmembers and estimating their fractional proportions, or abundances.
For decades, the standard workhorse of this field has been the Linear Mixing Model (LMM). The LMM operates on a simple geometric assumption: the observed spectrum is a weighted linear combination of the endmember signatures. While mathematically convenient and highly tractable, the linear model frequently fails in real-world scenarios. As comprehensive overviews like the one by Dobigeon et al. highlight, real-world hyperspectral data exhibits pervasive nonlinear mixing effects that distort abundance estimations if left unaddressed.
To understand why the linear model breaks down—and how modern research fixes it—we must look at the physical behavior of light.
Why the Linear Model Fails: The Physics of Mixing
The Linear Mixing Model assumes a "checkerboard" world: a flat surface where light hits a single material and bounces directly back to the sensor. In this idealized framework, a pixel that is 50% grass and 50% soil yields a spectrum that is exactly halfway between pure grass and pure soil.
In reality, earth and planetary surfaces are three-dimensional and microscopically complex. This gives rise to two primary types of nonlinearities:
- Multiple Scattering (Macroscopic Nonlinearity): Light does not always take a single path. In a forest canopy or an urban canyon, a photon might strike a leaf, reflect downward onto the soil, and then bounce upward toward the satellite sensor. As Chen et al. point out, this means the sensor captures light that has interacted with multiple substances sequentially. The resulting spectral signature is a product of these multiple interactions, introducing higher-order, cross-talk terms.
- Intimate Mixtures (Microscopic Nonlinearity): When different mineral grains or particulate materials are packed closely together—such as in planetary regolith or sand dunes—light reflects off multiple materials simultaneously at a microscopic scale. As explored by Nakhostin et al., this intimate mixing cannot be modeled by simple area-weighted averages because the scattering mechanics depend on the physical properties and geometries of the particles themselves.
Mathematical Frameworks for Nonlinear Unmixing
To capture these complex physical interactions, researchers have developed several distinct strategies to extend or bypass the linear model.
1. Polynomial and Linear-Quadratic Models
The most direct way to account for multiple scattering is to append higher-order interaction terms to the linear model. If $\mathbf{m}_i$ and $\mathbf{m}_j$ are the spectral vectors of two endmembers, a linear-quadratic (LQ) model introduces cross-product terms ($\mathbf{m}_i \odot \mathbf{m}_j$) to represent the secondary reflections between those two materials.
To solve this without exploding computational complexity, Kervazo et al. (EUSIPCO 2020) and Kervazo et al. (arxiv 2020) developed the Successive Nonnegative Projection Algorithm for Linear Quadratic Mixtures (SNPALQ). This framework provides a provably robust way to separate blind sources in near-separable mixtures, sequentially identifying the pure materials even when the data is muddied by second-order nonlinear cross-talk and noise.
2. Nonlinearity as Spatial Residuals or Outliers
Rather than explicitly modeling every possible interaction term, another powerful approach treats the linear model as a strong baseline and isolates nonlinearities as structured deviations.
- Spatial Residuals: Pereyra introduced a Bayesian framework that models pixels as a combination of a linear mix plus a spatial residual component. Because physical features like vegetation canopies or rocky outcroppings are spatially continuous, nonlinear effects are highly correlated among neighboring pixels. By incorporating a Gamma Markov Random Field (GMRF) and a positivity constraint on the residual terms, this method accurately detects and maps nonlinearities across an image.
- Sparse Outliers: Alternatively, Li et al. (2016) proposed Robust Collaborative Sparse Regression (RCSR). This method treats nonlinear mixing effects as sparse outliers. By using a joint sparse regression framework, the algorithm can absorb and isolate these non-linear distortions without needing a rigid, pre-defined physical model of the nonlinearity.
This philosophy of treating model deviations as structured anomalies is also vital in active remote sensing, such as multispectral Lidar. Works by Altmann et al. (2015), Altmann et al. (2016), and Altmann et al. (2017) demonstrate that sensor errors, unmodeled materials, and low-photon environmental interactions manifest as severe anomalies. By coupling linear unmixing with spatial-spectral anomaly detection (such as Ising models or GMRFs), they robustly isolate these non-ideal effects to preserve accurate material identification.
Data-Driven and Geometric Alternatives
When the exact physical nature of the mixing is entirely unknown or too complex for parametric equations, researchers turn to data-driven architectures or alternative geometries.
Deep Learning Autoencoders
Autoencoders are a natural fit for spectral unmixing because their bottleneck architecture mirrors the unmixing process: an encoder compresses the high-dimensional pixel spectrum into a low-dimensional representation (the abundances), and a decoder reconstructs the original spectrum from those abundances.
Observed Pixel ──> [ Encoder ] ──> Abundances (Bottleneck) ──> [ Decoder ] ──> Reconstructed Pixel
By utilizing nonlinear activation functions within the network, autoencoders can learn arbitrary, highly complex mixing physics without requiring prior knowledge of the environment. Dhaini et al. demonstrated this using an end-to-end convolutional autoencoder that extracts both endmembers and abundances simultaneously.
To prevent deep networks from acting as uninterpretable "black boxes," Li et al. (2022) designed a model-based deep autoencoder. This architecture explicitly embeds prior knowledge of the inverse relationship between the encoding and decoding processes, forcing the network to respect physical constraints while still capturing complex nonlinear interactions.
Optimal Transport
For challenging environments like planetary surfaces, where intimate mixing dominates, standard Euclidean distance metrics fail to capture how spectral shapes shift. Nakhostin et al. bypassed algebraic mixing models entirely by framing unmixing through the lens of Optimal Transport. By treating individual spectra as probability distributions and measuring the Wasserstein distance between them, their method tracks how light shifts across wavelengths, providing a highly robust framework for supervised planetary unmixing.
Summary
Moving beyond the Linear Mixing Model is essential for accurate remote sensing. Whether by adding polynomial interaction terms, isolating nonlinearities as spatial residuals, or leveraging the flexible representation power of deep autoencoders, modern nonlinear spectral unmixing bridges the gap between idealized linear geometry and the messy, multi-scattering reality of our physical world.
Key Foundational Work
Nonlinear Unmixing of Hyperspectral Images: Models and Algorithms. (2014) — contributed 3 concept elements to this direction.
Bayesian Nonlinear Hyperspectral Unmixing With Spatial Residual Component Analysis (2015) — contributed 1 concept elements to this direction.
Robust Linear Spectral Unmixing using Anomaly Detection (2015) — contributed 1 concept elements to this direction.