Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging

Fulin Luo, Xi Chen, Xiuwen Gong, Weiwen Wu, Tan GuoAAAI 2024 · DOI

TL;DR

This paper introduces the Dual-Window Multiscale Transformer (DWMT), a novel deep learning architecture for reconstructing hyperspectral images (HSIs) from compressed measurements. DWMT addresses the challenge of balancing long-range dependency capture with local feature extraction in Transformers for HSI, achieving state-of-the-art performance on both simulated and real-world data.

What Problem Does This Paper Address?

Hyperspectral imaging (HSI) captures rich spectral information across numerous contiguous bands, making it invaluable for applications like image classification and change detection. However, traditional hyperspectral cameras are expensive and slow. Snapshot compressive imaging (SCI) systems, particularly those using coded apertures (CASSI), offer a cost-effective and fast alternative by capturing a single 2D measurement and reconstructing the 3D HSI computationally. The core challenge in CASSI is solving the ill-posed inverse problem of HSI reconstruction from this limited 2D data. While recent Transformer-based methods have shown promise, they struggle to efficiently capture both the global spectral context and fine-grained spatial details without incurring prohibitive computational costs, especially given the high dimensionality of HSIs.

Methods

The proposed DWMT is an end-to-end Transformer-based framework that operates in two stages: coarse feature extraction and fine pixel refinement. Both stages utilize a U-Net architecture with full-scale skip connections.

The key innovation lies in the Dual-Window Multiscale Attention Block (DWMAB), which is central to the fine pixel refinement stage. DWMAB incorporates a novel Dual-Window Multiscale Multi-Head Self-Attention (DWM-MSA) mechanism. This mechanism divides the input into multiple branches that perform self-attention within two different-sized windows (e.g., 8x8 and 16x16). This "divide-and-conquer" approach allows the model to capture both local spatial details and longer-range dependencies within these windows, effectively balancing computational cost with representational power. Additionally, a Con-Abs Position Embedding (CAPE) method is introduced, which combines conditional and absolute position embeddings to better inform the model about spatial relationships.

The architecture also features a dual-branch encoder in the fine pixel refinement stage, where one branch handles downsampling and the other processes features through DWMABs. These branches are fused using a Cross Attention Fusion Module (CAFM).

Datasets

The paper evaluates DWMT on:

  • Simulated Data:
    • CAVE dataset: Used for training, consisting of 32 HSIs with a spatial size of 512x512.
  • KAIST dataset: Used for testing, comprising 30 HSIs with a spatial size of 2704x3376. Ten scenes from KAIST were used for testing.
  • Real Data:
    • Real HSIs captured by a CASSI system were used to validate the network's performance in a practical scenario.

The paper does not introduce new datasets but uses established benchmarks for comparison.

Key Insights and Ideas

  • Bridging the Gap in Transformers for HSI: The core insight is that a simple global or local Transformer is insufficient for HSI reconstruction. DWMT's dual-window approach effectively bridges this gap by allowing attention mechanisms to operate at multiple scales within local regions, capturing both fine details and broader context.
  • The Power of Multi-Scale Local Attention: The DWM-MSA mechanism demonstrates that by strategically partitioning attention computation across different window sizes, one can achieve a significant performance boost without the quadratic complexity of global attention. This is crucial for high-dimensional HSI data.
  • Position Matters (Even More): The CAPE method highlights the importance of robust positional encoding for Transformer-based vision tasks, especially when dealing with structured data like images. Combining conditional and absolute embeddings proves more effective than either alone.
  • Coarse-to-Fine for Reconstruction: The two-stage approach (coarse feature extraction followed by fine pixel refinement) is a sensible strategy for tackling complex reconstruction problems, allowing the model to first grasp the overall structure and then refine the details.

Key Results

  • Quantitative Performance: DWMT significantly outperforms existing methods, including CNN-based (TSA-Net, HDNet) and Transformer-based (MST, CST) approaches. On the KAIST dataset, DWMT achieved an average PSNR improvement of 1.64 dB over MST and 0.70 dB over CST, with corresponding SSIM improvements of 0.016 and 0.007.
  • Qualitative Results: Visualizations show that DWMT reconstructs clearer images with more accurate fine details, such as text on objects, compared to other methods. Spectral density curves also show higher correlation with ground truth, indicating better spectral consistency.
  • Ablation Studies: The paper presents ablation studies that confirm the crucial role of each proposed module: DWM-MSA, CAFM, and CAPE. Removing any of these components leads to a noticeable degradation in performance. The dual-window approach (using both 8x8 and 16x16 windows) also shows superior results compared to single-window approaches.

Limitations and Future Work

  • Computational Cost: While DWMT is more efficient than global Transformers, it still has a significant number of parameters and FLOPS compared to some simpler methods. The paper mentions parameter counts (14.48M) and FLOPS (46.71 G) for DWMT.
  • Generalization to Extreme Conditions: The paper demonstrates robustness on simulated noise and real-world data, but further exploration into extreme noise levels or different types of spectral distortions could be beneficial.
  • Future Work: The authors suggest that their framework can be extended to other SCI tasks. Further research could explore optimizing the window sizes, attention mechanisms, and positional embeddings for even greater efficiency and accuracy. Investigating the application of DWMT to other types of compressive sensing problems beyond HSI is also a potential avenue.

Summary updated

BibTeX
@inproceedings{luo2024dual,
  author = {Fulin Luo and Xi Chen and Xiuwen Gong and Weiwen Wu and Tan Guo},
  title = {{Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28190},
}
Figure