Multimodal Prior Learning with Double Constraint Alignment for Snapshot Spectral Compressive Imaging

Mingjin Zhang, Longyi Li, Fei Gao, Qiming Zhang, Jie GuoIJCAI 2025 · DOI

TL;DR

This paper introduces CAMM, a novel framework for snapshot spectral compressive imaging that leverages textual descriptions to improve reconstruction accuracy. By integrating fine-grained alignment of visual and textual features with a Mamba-based fusion mechanism, CAMM achieves state-of-the-art performance, addressing the limitations of existing methods that rely solely on image-level priors.

What Problem Does This Paper Address?

Snapshot spectral compressive imaging (SCI) aims to reconstruct a full 3D hyperspectral image (HSI) from a single 2D measurement. While traditional methods often focus on network architecture or simple image-level priors, they struggle to capture rich semantic information crucial for accurate reconstruction. Existing approaches, like those using dual cameras (DC-CASSI), often rely on RGB or grayscale images as priors, but these lack the detailed semantic understanding that textual descriptions can provide. A key challenge is bridging the gap between pre-trained vision-language models (VLMs) that understand RGB images and text, and the specific domain of HSI data. Furthermore, effectively fusing this multimodal information to guide the reconstruction process, especially considering the differing dimensionalities and complexities of features, remains an open problem.

Methods

The proposed framework, CAMM, consists of two main components:

  • Fine-grained Alignment Module (FAM): This module addresses the domain gap between pre-trained VLMs and HSI data. It employs a novel double-constraint approach that balances distribution similarity (using Bhattacharyya distance) and information entropy. This adaptive alignment strategy ensures that features are more accurately mapped, especially in regions with varying complexity. The alignment loss function, $L_{align}$, combines these constraints to achieve a more comprehensive feature mapping.
  • Multimodal Fusion Mamba (MFM): This component leverages the efficient global modeling capabilities of Mamba. In the encoder, MFM uses RGB image features to guide the Mamba's state updates, supplementing spatial information. In the decoder, it integrates both RGB image features and text features (processed by FAM) into Mamba's state-space equations. This integration is not a simple concatenation but a guided process where priors are incorporated at each scanning step, creating a positive feedback loop that enhances the accuracy of the reconstruction. The MFM utilizes a Rank-1 Tensor Generation (RTG) module and multi-head cross-attention (MHCA) to fuse multimodal information effectively.

Datasets

The authors used two public datasets for their experiments:

  • CAVE Dataset: A collection of HSIs.
  • KAIST Dataset: Another collection of HSIs.

For their experiments, they selected 10 scenes from KAIST (cropped to 256x256x28) and 10 scenes from CAVE (512x512x28). They also mention extending these datasets by generating textual data using GPT-4V, which was manually verified. Spectral interpolation was used to select 28 wavelengths (450-650 nm), and a two-pixel shift was applied to simulate spectral dispersion.

Key Insights and Ideas

  • Text as a Powerful Prior: The paper demonstrates that textual descriptions, when effectively integrated, can provide richer semantic information than image-level priors alone, significantly improving HSI reconstruction.
  • Bridging the VLM-HSI Gap: The FAM's double-constraint alignment (distribution similarity and entropy) is crucial for adapting general VLM knowledge to the specific domain of HSI.
  • Guided Reconstruction with Mamba: MFM's approach of integrating multimodal priors directly into Mamba's state-space equations, rather than simple feature fusion, allows for step-by-step guidance and a positive feedback mechanism, leading to more accurate reconstructions.
  • Adaptive Alignment: The FAM's ability to adaptively balance the influence of distribution similarity and information entropy allows it to handle both simple and complex feature regions effectively.

Key Results

  • Quantitative Performance: CAMM achieves state-of-the-art performance on the KAIST dataset, outperforming various CASSI and DC-CASSI based methods in terms of PSNR and SSIM. For example, on the KAIST dataset, CAMM achieved a PSNR of 49.25 and SSIM of 0.997, which were the best results among the compared methods.
  • Qualitative Improvements: Visual comparisons show that CAMM generates smoother textures, more precise edge details, and better spatial consistency of uniform regions compared to other methods. Spectral density curves also indicate superior spectral accuracy.
  • Ablation Studies: Removing FAM and MFM individually led to significant drops in PSNR, highlighting the importance of both modules. The FAM's proposed alignment method also outperformed simpler contrastive loss or Bhattacharyya distance-based alignment.

Limitations and Future Work

  • Computational Cost: While Mamba offers linear complexity, the overall framework's computational cost for training and inference is not explicitly detailed, though it is implied to be efficient compared to iterative unfolding networks.
  • Text Generation: The paper mentions using GPT-4V for text generation, which is a powerful tool, but the robustness and generalizability of text-based guidance across diverse scenes and object types could be further explored.
  • Future Directions: The authors suggest that their work opens avenues for exploring more sophisticated multimodal fusion strategies for HSI reconstruction, potentially incorporating other modalities or more advanced VLM architectures. The effectiveness of their approach in real-world applications beyond the tested datasets could also be a future research direction.

Summary updated

BibTeX
@inproceedings{zhang2025multimodal,
  author = {Mingjin Zhang and Longyi Li and Fei Gao and Qiming Zhang and Jie Guo},
  title = {{Multimodal Prior Learning with Double Constraint Alignment for Snapshot Spectral Compressive Imaging}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/263},
}
Figure