In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging

Xin Wang, Lizhi Wang, Xiangtian Ma, Maoqing Zhang, Lin Zhu, Hua Huang

Abstract

Dual-Camera Compressed Hyperspectral Imaging (DCCHI) enables the reconstruction of 3D Hyperspectral Images (HSI) by fusing compressive measurements with high-resolution Panchromatic (PAN) images. We introduce the Intra-Inter Similarity Exploiting Transformer (In2SET), a reconstruction network designed to maximize the utility of the PAN image. By utilizing intra-similarity within the PAN image as a proxy for HSI structure and aligning HSI features with PAN features for semantic consistency, In2SET significantly improves spatial-spectral fidelity. Integrated into a PAN-guided deep unrolling framework, our method outperforms state-of-the-art techniques in both reconstruction quality and computational efficiency across simulated and real-world datasets.

Introduction

Hyperspectral imaging (HSI) captures a three-dimensional data cube containing dense spectral information across hundreds of bands. Traditional snapshot methods like Coded Aperture Snapshot Spectral Imaging (CASSI) compress this 3D signal into a 2D measurement, necessitating complex reconstruction algorithms to solve the resulting ill-posed inverse problem. Dual-Camera Compressive Hyperspectral Imaging (DCCHI) enhances this process by incorporating a second camera to capture a high-resolution panchromatic (PAN) image. The challenge lies in effectively fusing these disparate data types. Previous approaches often relied on simple feature concatenation, failing to exploit the deep structural relationships between the sharp PAN image and the compressed HSI data.

Methods: PAN-guided Deep Unrolling (PGDU)

The proposed architecture utilizes a PAN-guided Deep Unrolling (PGDU) framework, which translates the iterative Half-Quadratic Splitting (HQS) optimization algorithm into a multi-stage neural network. Each stage of the network refines the HSI estimate, guided by a feature pyramid extracted from the high-resolution PAN image.

The PAN-guided Deep Unrolling (PGDU) framework, illustrating the multi-stage reconstruction pipeline guided by a PAN-derived feature pyramid.

At the heart of each stage is the In2SET Denoiser, a U-shaped Transformer that implements two key mechanisms:

  • Intra-Similarity Attention: This module uses the spatial textures of the PAN image as a proxy for the HSI's internal structure. It operates on the assumption that spatial similarity in the PAN domain correlates strongly with spectral similarity in the HSI domain.
  • Inter-Similarity Attention: Utilizing Cosine Similarity Reweighting (CRW), this module aligns features between the reconstructed HSI and the PAN image. It assigns higher weights to regions with strong semantic consistency, such as shared edges and textures, to refine the reconstruction.

Key Insights: The PAN Image as a Structural Proxy

The primary innovation of In2SET is the sophisticated use of the PAN image to guide the reconstruction process. Rather than treating the PAN image as mere additional input, the model exploits its inherent structural properties.

Conceptual illustration of Intra-Similarity and Inter-Similarity exploitation within the In2SET framework.
  • Proxy-based Guidance: Because a PAN image is an integral of spectral bands, its spatial features serve as a reliable map for the underlying HSI structure.
  • Semantic Alignment: The Inter-Similarity module ensures that the reconstructed HSI maintains semantic consistency with the sharp PAN image, effectively using the latter as a quality control mechanism.
  • Physical Consistency: By unrolling a physical imaging model, the framework ensures that the final output remains grounded in the physics of the DCCHI capture process.

Results and Evaluation

In2SET was evaluated on standard HSI benchmarks including CAVE, KAIST, and ICVL, as well as real-world data.

  • Superior Performance: The 9-stage In2SET model achieved a PSNR of 43.69 dB, surpassing the previous state-of-the-art (DAUHST) by over 1 dB.
  • Computational Efficiency: A 3-stage In2SET model provides performance comparable to a 5-stage DAUHST model while reducing computational costs by nearly 50% (20.78 GFLOPs vs. 40.50 GFLOPs).
  • Visual Fidelity: Reconstructions exhibit significantly sharper details and more accurate spectral curves, particularly in complex scenes.

Conclusion

In2SET advances the field of dual-camera compressive hyperspectral imaging by effectively leveraging the structural and semantic correlations between PAN and HSI data. The integration of intra-inter similarity exploitation within a deep unrolling framework provides a robust and efficient solution for high-fidelity HSI reconstruction. Future research will explore further optimizations to the initial reconstruction stages to enhance real-time processing capabilities.

Summary updated

Abstract

Dual-Camera Compressed Hyperspectral Imaging (DCCHI) offers the capability to reconstruct 3D Hyperspectral Image (HSI) by fusing compressive and Panchromatic (PAN) image, which has shown great potential for snapshot hyperspectral imaging in practice. In this paper, we introduce a novel DCCHI reconstruction network, the Intra-Inter Similarity Exploiting Transformer (In2SET). Our key insight is to make full use of the PAN image to assist the reconstruction. To this end, we propose using the intra-similarity within the PAN image as a proxy for approximating the intra-similarity in the original HSI, thereby offering an enhanced content prior for more accurate HSI reconstruction. Furthermore, we aim to align the features from the underlying HSI with those of the PAN image, maintaining semantic consistency and introducing new contextual information for the reconstruction process. By integrating In2SET into a PAN-guided unrolling framework, our method substantially enhances the spatial-spectral fidelity and detail of the reconstructed images, providing a more comprehensive and accurate depiction of the scene. Extensive experiments conducted on both real and simulated datasets demonstrate that our approach consistently outperforms existing state-of-the-art methods in terms of reconstruction quality and computational complexity. Code will be released.
BibTeX
@inproceedings{wang2023set,
  author = {Xin Wang and Lizhi Wang and Xiangtian Ma and Maoqing Zhang and Lin Zhu and Hua Huang},
  title = {{In2SET: Intra-Inter Similarity Exploiting Transformer for Dual-Camera Compressive Hyperspectral Imaging}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52733.2024.02350},
}
Figure