Dynamical Hyperspectral Unmixing with Variational Recurrent Neural Networks

Ricardo Augusto Borsoi, Tales Imbiriba, Pau Closas

Introduction

Hyperspectral imaging (HI) captures scenes across hundreds of narrow spectral bands, enabling precise material identification. However, the coarse spatial resolution of many satellite sensors often results in "mixed pixels" containing multiple materials. Hyperspectral unmixing (HU) aims to decompose these pixels into constituent materials, known as endmembers, and their corresponding fractional proportions, or abundances. A significant challenge in HU is Endmember Variability (EV), where the spectral signature of a material changes over time and space due to environmental factors such as seasonal shifts or atmospheric conditions.

Problem Statement

While multitemporal HI sequences offer rich information, existing unsupervised unmixing methods often struggle to simultaneously account for spatial and temporal variability. Supervised approaches exist but require extensive spectral libraries that are difficult to collect. The challenge lies in accurately unmixing a sequence of images and tracking material changes without prior "ground truth" labels or libraries.

The ReDSUNN Framework

The authors introduce ReDSUNN (Recurrent hyperSpectral Unmixing with Neural Networks), an unsupervised framework that treats multitemporal unmixing as a Bayesian inference problem. ReDSUNN integrates physical constraints with the flexibility of deep generative modeling through a Variational Recurrent Neural Network (VRNN).

The ReDSUNN architecture combines a stochastic generative model with a variational posterior to estimate abundances and endmembers from a sequence of hyperspectral images.

Key Innovations

  1. Smooth Generalized Linear Mixing Model (SGLMM): To manage EV, the authors propose a low-dimensional model where spectral variations are constrained to smooth basis functions using the Discrete Cosine Transform. This ensures physically realistic signatures while preventing overfitting.
  2. Stochastic Generative Model: The evolution of abundances and endmembers is modeled as a Markov process. Abundances are represented in a "softmax basis" to naturally enforce non-negativity and sum-to-one constraints.
  3. Variational RNN Posterior: A bidirectional LSTM approximates the intractable posterior distribution. This allows the model to leverage both past and future temporal context, improving state estimation for any given time step.
Graphical model representation of the dynamical unmixing process, contrasting a standard state-space model (left) with the proposed VRNN-based approach (right).

Experimental Evaluation

The researchers evaluated ReDSUNN on synthetic datasets and real-world data from Lake Tahoe.

  • Synthetic Results: In tests with Jasper Ridge spectral signatures, ReDSUNN consistently achieved lower Normalized Mean Squared Error (NRMSE) for both abundances and endmember signatures compared to state-of-the-art methods like Online Unmixing (OU) and Hierarchical Bayesian Unmixing (HBUN).
  • Real-World Robustness: On Lake Tahoe data acquired by the AVIRIS sensor, ReDSUNN successfully tracked seasonal changes in vegetation and water levels. It maintained a clear separation between soil and vegetation endmembers, whereas traditional methods often produced artifacts or confused the two materials.

Conclusion

ReDSUNN demonstrates that embedding physical models within deep learning architectures enables robust, unsupervised unmixing of complex temporal sequences. By leveraging bidirectional temporal context and smooth variability constraints, it provides a powerful tool for monitoring Earth's surface dynamics. Future work will focus on incorporating spatial correlations and automating hyperparameter tuning.

Summary updated

BibTeX
@article{borsoi2023dynamical,
  author = {Ricardo Augusto Borsoi and Tales Imbiriba and Pau Closas},
  title = {{Dynamical Hyperspectral Unmixing with Variational Recurrent Neural Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  url = {https://openreview.net/forum?id=8Hg8CCfh64},
}
Figure