Abstract
Dual-Camera Compressed Hyperspectral Imaging (DCCHI) enables the reconstruction of 3D Hyperspectral Images (HSI) by fusing compressive measurements with high-resolution Panchromatic (PAN) images. We introduce the Intra-Inter Similarity Exploiting Transformer (In2SET), a reconstruction network designed to maximize the utility of the PAN image. By utilizing intra-similarity within the PAN image as a proxy for HSI structure and aligning HSI features with PAN features for semantic consistency, In2SET significantly improves spatial-spectral fidelity. Integrated into a PAN-guided deep unrolling framework, our method outperforms state-of-the-art techniques in both reconstruction quality and computational efficiency across simulated and real-world datasets.
Introduction
Hyperspectral imaging (HSI) captures a three-dimensional data cube containing dense spectral information across hundreds of bands. Traditional snapshot methods like Coded Aperture Snapshot Spectral Imaging (CASSI) compress this 3D signal into a 2D measurement, necessitating complex reconstruction algorithms to solve the resulting ill-posed inverse problem. Dual-Camera Compressive Hyperspectral Imaging (DCCHI) enhances this process by incorporating a second camera to capture a high-resolution panchromatic (PAN) image. The challenge lies in effectively fusing these disparate data types. Previous approaches often relied on simple feature concatenation, failing to exploit the deep structural relationships between the sharp PAN image and the compressed HSI data.
Methods: PAN-guided Deep Unrolling (PGDU)
The proposed architecture utilizes a PAN-guided Deep Unrolling (PGDU) framework, which translates the iterative Half-Quadratic Splitting (HQS) optimization algorithm into a multi-stage neural network. Each stage of the network refines the HSI estimate, guided by a feature pyramid extracted from the high-resolution PAN image.
At the heart of each stage is the In2SET Denoiser, a U-shaped Transformer that implements two key mechanisms:
- Intra-Similarity Attention: This module uses the spatial textures of the PAN image as a proxy for the HSI's internal structure. It operates on the assumption that spatial similarity in the PAN domain correlates strongly with spectral similarity in the HSI domain.
- Inter-Similarity Attention: Utilizing Cosine Similarity Reweighting (CRW), this module aligns features between the reconstructed HSI and the PAN image. It assigns higher weights to regions with strong semantic consistency, such as shared edges and textures, to refine the reconstruction.
Key Insights: The PAN Image as a Structural Proxy
The primary innovation of In2SET is the sophisticated use of the PAN image to guide the reconstruction process. Rather than treating the PAN image as mere additional input, the model exploits its inherent structural properties.
- Proxy-based Guidance: Because a PAN image is an integral of spectral bands, its spatial features serve as a reliable map for the underlying HSI structure.
- Semantic Alignment: The Inter-Similarity module ensures that the reconstructed HSI maintains semantic consistency with the sharp PAN image, effectively using the latter as a quality control mechanism.
- Physical Consistency: By unrolling a physical imaging model, the framework ensures that the final output remains grounded in the physics of the DCCHI capture process.
Results and Evaluation
In2SET was evaluated on standard HSI benchmarks including CAVE, KAIST, and ICVL, as well as real-world data.
- Superior Performance: The 9-stage In2SET model achieved a PSNR of 43.69 dB, surpassing the previous state-of-the-art (DAUHST) by over 1 dB.
- Computational Efficiency: A 3-stage In2SET model provides performance comparable to a 5-stage DAUHST model while reducing computational costs by nearly 50% (20.78 GFLOPs vs. 40.50 GFLOPs).
- Visual Fidelity: Reconstructions exhibit significantly sharper details and more accurate spectral curves, particularly in complex scenes.
Conclusion
In2SET advances the field of dual-camera compressive hyperspectral imaging by effectively leveraging the structural and semantic correlations between PAN and HSI data. The integration of intra-inter similarity exploitation within a deep unrolling framework provides a robust and efficient solution for high-fidelity HSI reconstruction. Future research will explore further optimizations to the initial reconstruction stages to enhance real-time processing capabilities.
Summary updated