Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach

Charilaos I. Kanatsoulis, Xiao Fu, Nicholas D. Sidiropoulos, Wing-Kin MaIEEE TSP 2018 · OpenReview

Abstract

Hyperspectral super-resolution (HSR) aims to fuse a hyperspectral image (HSI) and a multispectral image (MSI) to produce a super-resolution image (SRI) with high spatial and spectral resolutions. Traditional methods often rely on low-rank matrix approximations, which can lead to a loss of structural information and lack theoretical guarantees for identifiability. This paper proposes a coupled tensor factorization framework that treats images as third-order tensors. This approach ensures the identifiability of the SRI under realistic conditions and remains effective even when degradation operators are unknown.

Introduction: The Challenge of Image Fusion

Hyperspectral images (HSI) are characterized by high spectral resolution but often suffer from poor spatial resolution. Multispectral images (MSI), on the other hand, offer sharp spatial detail but limited spectral information. The goal of HSR is to combine these complementary data sources into a single Super-Resolution Image (SRI).

Fusing HSI and MSI to produce a high-resolution SRI.

Current state-of-the-art methods face three primary hurdles:

  1. Structural Loss: Flattening 3D image cubes into 2D matrices discards inherent spatial-spectral correlations.
  2. Identifiability: It is often unclear if matrix-based fusion can uniquely recover the ground-truth image.
  3. Blind Degradation: Most algorithms require precise knowledge of the spatial blurring and spectral downsampling processes, which are rarely known in practice.

The Tensor Factorization Approach

To address these issues, the proposed framework models the SRI as a third-order tensor and applies Canonical Polyadic Decomposition (CPD). This decomposition represents a tensor as a sum of rank-one components, aligning with the Linear Mixture Model (LMM) where pixels are combinations of material "endmembers."

Mathematical modeling of the degradation process from SRI to HSI and MSI.

The degradation process is modeled as linear operators acting on the different modes of the tensor. The authors introduce the STEREO algorithm, a joint optimization framework that minimizes reconstruction error for both HSI and MSI. A "Blind" version of STEREO is also developed, which exploits the uniqueness of tensor decompositions to identify latent factors without prior knowledge of the spatial degradation operator.

Theoretical Guarantees and Identifiability

A key contribution of this work is the formal proof of identifiability. Unlike matrix factorization, which often requires heavy constraints like sparsity to find a unique solution, tensor CPD is essentially unique under mild conditions. The paper provides theorems showing that the SRI can be recovered almost surely if its rank falls within specific bounds, providing a rigorous mathematical foundation for the fusion process.

Experimental Evaluation

The approach was tested on four benchmark datasets: SALINAS, Cuprite, Indian Pines, and Pavia University. Using "semi-real" simulations where high-resolution HSIs were synthetically degraded, the researchers compared STEREO against several baselines.

  • Accuracy: On the SALINAS dataset, STEREO achieved a Reconstruction Signal-to-Noise Ratio (R-SNR) of 38.62 dB, significantly outperforming matrix-based methods.
  • Robustness: Blind STEREO maintained high performance even when blurring kernels were unknown or misaligned, a scenario where traditional methods like HySure and CNMF struggled.
  • Efficiency: The tensor-based algorithms demonstrated runtimes comparable to existing iterative matrix methods, making them practical for real-world applications.

Conclusion

By shifting from a matrix-centric to a tensor-centric perspective, this research provides a more robust and theoretically sound framework for hyperspectral super-resolution. The ability to guarantee identifiability and operate "blindly" regarding degradation operators marks a significant advancement in the field. Future work will focus on automated rank selection and extending the model to handle non-linear mixing effects in complex environments.

Summary updated

Abstract

Hyperspectral super-resolution refers to the problem of fusing a hyperspectral image (HSI) and a multispectral image (MSI) to produce a super-resolution image (SRI) that admits fine spatial and spectral resolutions. State-of-the-art methods approach the problem via low-rank matrix approximations to the matricized HSI and MSI. These methods are effective to some extent, but a number of challenges remain. First, HSIs and MSIs are naturally third-order tensors (data “cubes”) and thus matricization is prone to a loss of structural information, which could degrade performance. Second, it is unclear whether these low-rank matrix-based fusion strategies can guarantee the identifiability of the SRI under realistic assumptions. However, identifiability plays a pivotal role in estimation problems and usually has a significant impact on practical performance. Third, a majority of the existing methods assume known (or easily estimated) degradation operators from the SRI to the corresponding HSI and MSI, which is hardly the case in practice. In this paper, we propose to tackle the super-resolution problem from a tensor perspective. Specifically, we utilize the multidimensional structure of the HSI and MSI to propose a coupled tensor factorization framework that can effectively overcome the aforementioned issues. The proposed approach guarantees the identifiability of the SRI under mild and realistic conditions. Furthermore, it works with little knowledge about the degradation operators, which is clearly a favorable feature in practice. Semi-real scenarios are simulated to showcase the effectiveness of the proposed approach.
BibTeX
@article{kanatsoulis2018hyperspectral,
  author = {Charilaos I. Kanatsoulis and Xiao Fu and Nicholas D. Sidiropoulos and Wing-Kin Ma},
  title = {{Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach}},
  journal = {IEEE Trans. Signal Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=Kmp4tQ-BjoL},
}
Figure