A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration

Théo Bodrito, Alexandre Zouaoui, Jocelyn Chanussot, Julien Mairal

Abstract

Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where deep models tend to shine. In this paper, we advocate instead for a hybrid approach based on sparse coding principles that retains the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. We show on various denoising benchmarks that our method is computationally efficient and significantly outperforms the state of the art.

Introduction

Hyperspectral imaging (HSI) captures a scene across hundreds of narrow spectral bands, providing a unique "spectral signature" for every pixel. This capability is invaluable for fields like precision farming and astrophysics. However, HSI data is frequently degraded by complex noise—including Gaussian noise, thermal noise, and sensor-specific artifacts like stripes.

A significant challenge in HSI restoration is the scarcity of clean ground-truth data. Unlike standard RGB photography, large-scale datasets of noise-free hyperspectral images are non-existent. This makes training traditional deep learning models, such as CNNs, difficult and prone to overfitting. Historically, researchers have relied on learning-free classical methods that use handcrafted priors (e.g., low-rank or sparsity). While interpretable and data-efficient, these methods are often slow and difficult to tune.

Comparison of learning-free and learning-based restoration paradigms.

To address these issues, we propose T3SC (Trainable Spectral-Spatial Sparse Coding), a hybrid model that combines the interpretability of sparse coding with the learning power of deep neural networks.

Methodology: The T3SC Architecture

The T3SC model is designed as a two-layer architecture that mimics the physical properties of hyperspectral data.

The T3SC architecture, featuring spectral and spectral-spatial sparse coding layers.

1. Spectral Sparse Coding (Layer 1)

This layer processes each pixel individually, decomposing its spectrum into a sparse combination of basis elements. This effectively performs automated "spectral unmixing," identifying the underlying materials within a single pixel.

2. Spectral-Spatial Sparse Coding (Layer 2)

To capture spatial relationships and textures, this layer operates on patches of pixels. It leverages the spatial redundancy inherent in natural scenes to further refine the denoising process.

3. Key Innovations

  • Low-Rank Dictionaries: The dictionaries (basis elements) are constrained to be low-rank, which keeps the model lightweight and reduces the number of parameters to learn.
  • Unrolled Optimization (LISTA): Instead of a generic CNN, the model "unrolls" the classical Iterative Soft-Thresholding Algorithm (ISTA) into a fixed number of layers. This makes the entire process differentiable, allowing for end-to-end training via backpropagation.
  • Noise Adaptation: A small sub-network estimates band-wise noise levels, allowing the model to dynamically adjust its denoising strength at test time.
  • Self-Supervised Learning (T3SC-SSL): For scenarios without ground truth, we introduce a "blind-band" approach where the model learns to reconstruct randomly masked spectral bands from the remaining ones.

Experiments and Results

We evaluated T3SC on several standard HSI benchmarks, including ICVL (high-resolution images with 31 bands) and the Washington DC Mall (remote sensing data with 191 bands).

Performance and Efficiency

T3SC consistently outperformed or matched state-of-the-art supervised and unsupervised methods (such as BM4D, GLF, and QRNN3D) across various noise patterns, including Gaussian noise and stripes. Notably, T3SC is significantly faster than high-performing classical methods. For instance, on the ICVL dataset, T3SC takes approximately 5.8 seconds per image, whereas the classical NGMeet method requires 2686 seconds.

Interpretability and Robustness

The sparse coding foundation provides inherent interpretability: the first layer maps different sensors to a common feature space, while the second layer handles spatial denoising. Furthermore, the internal noise estimator makes the model robust to "blind" denoising tasks where noise levels are unknown beforehand.

Conclusion

T3SC demonstrates that hybridity is a powerful strategy for HSI restoration. By embedding domain knowledge into a trainable architecture, we achieve high performance with limited data. While the self-supervised version (T3SC-SSL) still faces challenges with complex, non-Gaussian noise, the overall framework provides a computationally efficient and interpretable path forward for hyperspectral imaging. Future work will focus on making the pipeline more sensor-agnostic and improving self-supervised handling of structured noise like stripes.

Summary updated

Abstract

Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where deep models tend to shine. In this paper, we advocate instead for a hybrid approach based on sparse coding principles that retains the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. We show on various denoising benchmarks that our method is computationally efficient and significantly outperforms the state of the art.
BibTeX
@inproceedings{bodrito2021trainable,
  author = {Théo Bodrito and Alexandre Zouaoui and Jocelyn Chanussot and Julien Mairal},
  title = {{A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bycXC47nkd},
}
Figure