A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration

Théo Bodrito, Alexandre Zouaoui, Jocelyn Chanussot, Julien Mairal

TL;DR

Hyperspectral image (HSI) denoising is challenging because deep learning models lack massive "clean" datasets and struggle with varying sensor types. This paper introduces T3SC, a hybrid model combining the interpretability of classical sparse coding with end-to-end deep learning. By baking physical priors into the architecture, it achieves state-of-the-art denoising performance with high computational efficiency, even offering a self-supervised mode that requires no clean ground-truth data.

What Problem Does This Paper Address?

Hyperspectral imaging captures the electromagnetic spectrum across dozens or hundreds of bands per pixel. This rich data is invaluable for fields like remote sensing, agriculture, and medical imaging, as it can identify the specific material signatures of objects. However, HSI sensors are prone to complex degradation, including thermal noise, photon noise, and structured "stripe" noise.

Comparison of learning-free and learning-based methods for hyperspectral image restoration.

Denoising is a crucial first step, but standard deep learning (which dominates RGB image restoration) struggles here for two main reasons:

  1. Data Scarcity: There is a severe lack of large-scale, high-quality "clean" ground-truth hyperspectral data to train supervised neural networks.
  2. Sensor Diversity: Different sensors capture a different number of spectral bands, making it hard to design a one-size-fits-all neural network architecture.

Traditional, non-learning methods (like BM4D) don't need training data, but they are computationally agonizing at inference time. The authors aim to bridge this gap with a fast, trainable model that doesn't require massive datasets.

Methods

The authors propose T3SC (Trainable Spectral-Spatial Sparse Coding), a hybrid approach that uses "differentiable programming." Instead of a standard Convolutional Neural Network (CNN), they take a classical optimization algorithm for sparse coding (LISTA) and "unroll" its iterations into the layers of a neural network. This allows the model to be trained end-to-end via backpropagation.

Architecture of the T3SC model showing the Spectral and Spectral-Spatial Sparse Coding layers.

The architecture is split into two distinct layers to handle sensor diversity:

  • Layer 1 (Spectral): This layer is sensor-specific. It looks at individual pixels (1x1 patches) and unmixes the spectral signature using a learned dictionary. It maps the arbitrary number of input bands into a common feature space.
  • Layer 2 (Spectral-Spatial): This layer is shared/generic. It looks at 2D spatial patches to capture local relationships between neighboring pixels.

To further reduce the number of parameters (preventing overfitting on small datasets), they enforce a low-rank structure on the learned dictionaries, assuming that the spectral and spatial domains are nearly separable.

Finally, they introduce two powerful variants:

  1. Noise-Adaptive: A module that learns to estimate and adjust for different noise variances across different spectral bands.
  2. Self-Supervised Learning (SSL): A "blind-band" training scheme where the model masks one spectral band and tries to reconstruct it using the others, allowing the model to train without any clean ground-truth images.

Datasets

The model was evaluated on standard HSI datasets representing different domains:

  • ICVL: A close-range dataset with 31 bands.
  • Washington DC Mall: A remote sensing dataset with 191 bands.
  • Urban, APEX, Pavia, and CAVE: Used for additional experiments and real-world noise testing.

Key Insights and Ideas

  • Priors > Massive Data: When training data is scarce, baking domain knowledge directly into the network architecture is incredibly powerful. By enforcing sparsity and low-rank structures, T3SC achieves excellent results with a fraction of the parameters used by standard deep learning models.
  • The Two-Layer Sensor Solution: The split architecture elegantly solves the "different sensors = different bands" problem. By using a lightweight, sensor-specific first layer to project data into a common space, the heavier, more complex spatial-spectral second layer can be shared across entirely different datasets.
  • Interpretability Matters: Because T3SC is based on physical principles (linear spectral unmixing), its internal workings are understandable. This interpretability made it easy for the authors to inject a principled noise-adaptive weighting scheme, something that is much harder to do in a "black box" CNN.

Key Results

  • Quantitative Superiority: T3SC achieves state-of-the-art results on the ICVL dataset across various noise types. For example, under Gaussian noise ($\sigma=25$), T3SC achieved an MPSNR of 45.38, beating the SOTA deep learning model QRNN3D (44.20) and the best traditional method NGMeet (44.74).
  • Computational Efficiency: T3SC is vastly faster than traditional methods. It processes an ICVL image in 5.8 seconds. By comparison, traditional methods like GLF take ~5,500 seconds, and LLRT takes over 24,000 seconds. T3SC's speed is highly competitive with fast deep models like QRNN3D (3.6s).
  • Real-World Transfer: The authors demonstrated that T3SC, trained only on synthetic noise, successfully transfers to denoising real-world sensor noise on the Urban dataset.

Limitations and Future Work

The authors acknowledge a limitation regarding their Self-Supervised Learning (SSL) variant. While the "blind-band" approach performs quite well on simple, independent (i.i.d.) Gaussian noise, its performance drops when faced with more complex, structured noise (like sensor stripes). Overcoming this limitation will require designing new, more sophisticated self-supervised learning schemes tailored to the unique physical realities of hyperspectral degradation.

Summary updated

Abstract

Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where deep models tend to shine. In this paper, we advocate instead for a hybrid approach based on sparse coding principles that retain the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. We show on various denoising benchmarks that our method is computationally efficient and significantly outperforms the state of the art.
BibTeX
@inproceedings{bodrito2021trainable,
  author = {Théo Bodrito and Alexandre Zouaoui and Jocelyn Chanussot and Julien Mairal},
  title = {{A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bycXC47nkd},
}
Figure