Coarse-Fine Spectral-Aware Deformable Convolution for Hyperspectral Image Reconstruction

Jincheng Yang, Lishun Wang, Miao Cao, Huan Wang, Yinping Zhao, Xin Yuan

Introduction

Hyperspectral Imaging (HSI) provides a rich 3D data cube containing spatial and spectral information, essential for applications ranging from medical diagnostics to remote sensing. However, capturing this data traditionally requires time-consuming scanning. Coded Aperture Snapshot Spectral Imaging (CASSI) addresses this by compressing the 3D cube into a single 2D snapshot. The primary challenge lies in the inverse problem: reconstructing the high-dimensional original cube from the compressed measurement.

Recent advancements have seen a shift from traditional CNNs, which suffer from limited receptive fields, to Transformer-based models that capture global context. However, Transformers introduce significant computational overhead. This paper introduces the Coarse-Fine Spectral-Aware Deformable Convolution Network (CFSDCN), an architecture designed to combine the efficiency of CNNs with the long-range modeling capabilities of Transformers.

The CASSI Reconstruction Challenge

In a CASSI system, the 3D HSI cube is modulated by a physical mask and then dispersed by a prism before being integrated into a 2D sensor. This process "smears" the spectral information across the spatial dimensions. Reconstructing the original cube requires a model that can understand non-local spatial similarities and complex spectral correlations.

Existing CNNs often fail because their fixed, local kernels cannot capture the long-range dependencies inherent in HSI data. Conversely, while Transformers excel at global modeling, their quadratic computational complexity makes them impractical for real-time or edge deployment. CFSDCN bridges this gap by utilizing deformable convolutions as a lightweight alternative to self-attention.

Proposed Method: CFSDCN

The CFSDCN architecture follows a U-Net style structure (Encoder-Bottleneck-Decoder) and introduces two key innovations to handle the unique properties of hyperspectral data.

Grouped Deformable Convolutions (DCN)

For the first time in CASSI reconstruction, the authors employ Deformable Convolutions. Unlike standard convolutions that sample from a rigid grid, DCN learns offsets to sample from flexible, data-dependent locations. This allows the network to "reach out" and capture non-local similarities, effectively mimicking the benefits of a Transformer's attention mechanism with the efficiency of a convolutional operation.

Coarse-Fine Spectral-Aware Block (CFSAB)

To exploit the strong correlations across spectral channels, the authors designed the CFSAB. This module processes spectral information at two granularities:

  1. Fine-grained: Uses point-wise and depth-wise convolutions to capture immediate, pixel-level spectral similarities.
  2. Coarse-grained (LCS Module): Utilizes larger $7 \times 7$ kernels and dimensionality reduction to capture broader, structural spectral patterns across different positions and channels.
The CFSDCN architecture, featuring the Coarse-Fine Spectral-Aware Block (CFSAB) and Deformable Convolution (CFSDConv).

Experimental Results

The model was evaluated on the CAVE and KAIST datasets for simulation, as well as on real-world data from a physical CASSI system.

Performance and Efficiency

CFSDCN achieves state-of-the-art (SOTA) results while maintaining a significantly lower computational footprint than competing models. The "Large" version (CFSDCN-L) achieved a PSNR of 36.31 dB, outperforming the previous SOTA Transformer model (CST-L). More impressively, the "Small" version (CFSDCN-S) uses only 0.76M parameters and 9.45 GFLOPs, yet it surpasses much larger models like TSA-net (44.25M parameters).

Performance vs. Efficiency: PSNR (dB) vs. FLOPs (G) for various HSI reconstruction methods.

Visual Quality

Qualitative analysis demonstrates that CFSDCN produces reconstructions with sharper edges and fewer chromatic aberrations. By effectively finding sparse, relevant information in the 2D snapshot, the deformable kernels preserve fine textures that fixed-kernel CNNs often blur.

Conclusion

CFSDCN represents a significant step forward in HSI reconstruction by demonstrating that deformable convolutions can serve as a highly efficient proxy for self-attention. By capturing both coarse and fine spectral relationships, the model achieves superior accuracy with a fraction of the parameters required by Transformer-based architectures. Future work may involve testing the model's robustness against complex hardware non-linearities in real-world satellite or medical imaging systems.

Summary updated

BibTeX
@inproceedings{yang2024coarse,
  author = {Jincheng Yang and Lishun Wang and Miao Cao and Huan Wang and Yinping Zhao and Xin Yuan},
  title = {{Coarse-Fine Spectral-Aware Deformable Convolution for Hyperspectral Image Reconstruction}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647725},
}
Figure