TL;DR
This paper introduces a novel approach for fusing hyperspectral and multispectral images acquired through compressive sensing. By incorporating a "non-local low-rank" prior on the abundance maps, the method significantly improves the spatial and spectral resolution of the fused image compared to existing techniques, even when using a fraction of the data.
What Problem Does This Paper Address?
Compressive spectral imaging (CSI) offers a way to acquire spectral images (SIs) with reduced data, storage, and processing costs. However, reconstructing high-quality SIs from these compressed measurements is challenging, especially when fusing information from two different compressed sources: a hyperspectral image (HSI) with high spectral but low spatial resolution, and a multispectral image (MSI) with high spatial but low spectral resolution. This task, known as compressive spectral image fusion (CSIF), aims to produce a high spatial-spectral resolution SI. Existing methods often rely on priors like global low-rank, smoothness, or sparsity. While a "non-local low-rank" (NLLR) prior has shown promise for other CSI problems by exploiting self-similarities within image patches, it hadn't been applied to CSIF. This paper fills that gap by proposing a CSIF method that jointly leverages global low-rank, smoothness, and this NLLR prior on the abundance maps.
Methods
The core of the proposed method lies in its mathematical formulation and the algorithm used to solve it.
- Linear Mixture Model (LMM): The paper models the spectral image as a linear combination of a few "endmembers" (spectral signatures) and their corresponding "abundances" (fractional proportions). This LMM implicitly captures the global low-rank property of SIs.
- Non-Local Low-Rank (NLLR) Prior on Abundances: Instead of applying the NLLR prior to the entire image, the authors apply it to the abundance maps. This is a key innovation that reduces computational complexity. The NLLR prior is implemented through a four-step process:
- Patch Extraction: Dividing the abundance maps into small, overlapping spatial patches.
- Patch Grouping: Grouping similar patches together based on a similarity metric.
- Low-Rank Approximation: Finding a low-rank approximation for each group of patches.
- Aggregation: Averaging the estimations from the low-rank approximations to reconstruct the final abundance map.
- Optimization Problem: The CSIF problem is formulated as an optimization problem that minimizes a cost function including:
- Data Fidelity Terms: Ensuring the reconstructed image matches the compressed measurements from both HSI and MSI.
- Smoothness Prior (Total Variation): Promoting smooth transitions in the abundance maps while preserving sharp edges.
- NLLR Prior: Exploiting the self-similarities within patches of the abundance maps.
- ADMM Algorithm: The optimization problem is solved using the Alternating Direction Method of Multipliers (ADMM), a powerful iterative technique for solving complex optimization problems.
Datasets
The authors used modified versions of three publicly available hyperspectral image databases for evaluation:
- Jasper F1: A subregion of the Jasper F1 database.
- Urban F2: A subregion of the Urban database.
- Pavia University F3: A subregion of the Pavia University database.
These databases were used to simulate compressed HSI and MSI data by applying spatial and spectral degradation factors, mimicking the compressive sensing acquisition process.
Key Insights and Ideas
- NLLR on Abundances is Key: Applying the NLLR prior to the abundance maps, rather than the full spectral image, is a crucial insight that significantly reduces computational burden while effectively leveraging spatial self-similarities.
- Synergistic Priors: The combination of global low-rank (via LMM), smoothness (TV), and NLLR priors leads to superior fusion performance. Each prior contributes a different aspect of the spectral image's structure.
- Efficiency of Compressive Sensing: The results demonstrate that even with a small fraction of the data (e.g., 7.8% to 100% of the HRI), the proposed method can achieve comparable or even better results than methods using full data, highlighting the power of compressive sensing when combined with effective reconstruction techniques.
- Visual Similarity: The NLLR prior leads to a better visual similarity between the reconstructed spectral responses and the original ones, suggesting a more accurate representation of the scene's spectral characteristics.
Key Results
The proposed NLLR-based CSIF method consistently outperformed existing CSIF methods (based on sparsity prior) and a full-resolution fusion method (FUMI) across various metrics:
- PSNR: An average improvement of 6.8 dB was observed.
- UIQI: An average gain of 0.02 was achieved.
- SAM: A reduction of 0.5 in the spectral angle mapper metric.
- ERGAS: A reduction of 1.5 in the dimensionless global relative error of synthesis metric.
Qualitative results also showed that the fused images obtained with the NLLR prior were visually closer to the original images, particularly in terms of spectral reconstruction.
Limitations and Future Work
- Computational Cost: While the NLLR prior on abundances reduces complexity, the overall computational cost is still high, especially for the SVD computations within the NLLR step. The paper notes that this can be mitigated by fixing patch-group indices for some iterations.
- Parameter Tuning: The performance of the algorithm is sensitive to the choice of regularization parameters (μ, τ, and ρ). The paper suggests cross-validation for parameter selection and shows that μ, controlling the NLLR approximation, is particularly influential.
- Non-Convexity: The NLLR prior introduces non-convexity into the optimization problem, which can affect global convergence. The authors suggest that local convergence can be achieved by fixing patch-group assignments for a few iterations.
- Future Work: The paper implicitly suggests further research into optimizing the ADMM algorithm for faster convergence and exploring more advanced techniques for patch grouping and low-rank approximation to further reduce computational complexity. Investigating the application of this NLLR prior to other CSI-related problems also remains an open avenue.
Summary updated