Multiscale Adaptive Fusion Network for Hyperspectral Image Denoising

Haodong Pan, Feng Gao, Junyu Dong, Qian Du

TL;DR

This paper introduces MAFNet, a novel deep learning network for hyperspectral image denoising that effectively tackles the challenge of preserving fine spatial details while capturing broad contextual information. By employing a multi-scale fusion strategy and a co-attention mechanism, MAFNet achieves state-of-the-art denoising performance on various datasets, outperforming existing methods.

What Problem Does This Paper Address?

Hyperspectral images (HSIs) are invaluable for applications like land cover classification and change detection. However, the image acquisition process often introduces noise, significantly degrading their visual quality and hindering downstream analysis. Removing this noise (denoising) is a critical preprocessing step. Existing methods struggle with a fundamental trade-off: preserving fine spectral and spatial details while simultaneously capturing the broader contextual information present in HSIs. This is further complicated by the inherent multiscale nature of image information, where correlations across different scales are not always effectively exploited by current techniques.

Methods

The proposed Multi-scale Adaptive Fusion Network (MAFNet) addresses these challenges through a novel architecture:

  • Multi-scale Information Aggregation: MAFNet first generates downsampled versions of the input HSI. These multiscale representations are then processed through a coarse-fusion network. This network focuses on transferring information from lower-resolution features to higher-resolution ones, aiming to expand the receptive field and capture more contextual information.
  • Fine-Scale Fusion and Co-Attention: A subsequent fine-fusion network further integrates information across parallel multiscale subnetworks. The core innovation here is the co-attention fusion module. Unlike simpler fusion methods (like concatenation or element-wise summation), this module adaptively emphasizes informative features from different scales using learned weights. This allows for a more discriminative feature representation.
  • Adaptive Instance Normalization (AIN): Within the coarse-fusion network, AIN modules are used to transfer semantic information from downscaled features to higher-resolution ones. This module learns affine transformation parameters (shift and scale) from downscaled features to adaptively normalize and enhance the current feature map.
  • Loss Function: The network is optimized using a combination of L1 reconstruction loss and a global gradient regularizer to preserve image details.

Datasets

The paper uses several datasets for training and evaluation:

  • Training: ICVL and CAVE datasets were used for training.
  • Evaluation:
    • ICVL dataset (31 spectral bands, 1392x1300 pixels)
    • CAVE dataset (31 spectral bands, 512x512 pixels)
    • Pavia University dataset (103 bands, 610x340 pixels)
    • Urban dataset (captured by HYDICE sensor, 210 bands)
    • Indian Pines dataset (captured by AVIRIS sensor, 220 bands)

The paper also defines five complex noise scenarios for testing, ranging from non-i.i.d. Gaussian noise to mixtures of Gaussian, stripe, deadline, and impulse noise.

Key Insights and Ideas

  • Multiscale Fusion is Key: The paper strongly advocates for the explicit processing and fusion of information across multiple scales. This is crucial for balancing local detail preservation with global context understanding.
  • Adaptive Feature Emphasis: The co-attention fusion module is a significant contribution. It moves beyond simple feature aggregation by learning to dynamically weigh the importance of features from different scales, leading to more robust and discriminative representations.
  • AIN for Semantic Transfer: The Adaptive Instance Normalization (AIN) module is an effective way to transfer contextual semantic information from lower-resolution feature maps to higher-resolution ones, aiding in the preservation of structural details.
  • Incremental Learning for Robustness: The use of an incremental learning strategy, starting with simpler noise and progressing to more complex noise types, helps the model build robustness and achieve better generalization.

Key Results

  • State-of-the-Art Performance: MAFNet consistently outperforms several state-of-the-art denoising methods (including traditional methods like BM4D, LRMR, LRTV, and deep learning methods like HSID-CNN, MemNet, QRNN3D) across various quantitative metrics (PSNR, SSIM, SAM) on both synthetic and real HSI datasets.
  • Superior Denoising: For example, on the ICVL dataset with σ=30 Gaussian noise, MAFNet achieved a PSNR of 43.97 dB, compared to 43.46 dB for QRNN3D.
  • Preservation of Details: Visual results demonstrate that MAFNet effectively removes noise while preserving fine spatial textures and spectral details, often outperforming baselines that tend to lose such information or introduce artifacts.
  • Robustness to Complex Noise: The method shows strong performance even under complex noise conditions, which are more challenging for existing techniques.

Limitations and Future Work

  • Computational Complexity: While MAFNet achieves excellent performance, the paper acknowledges that its computational complexity can be higher than some simpler methods, especially for larger model variants (MAFNet-L).
  • Lightweight Techniques: The authors suggest that future work will focus on developing more lightweight HSI denoising techniques, potentially by exploring more efficient feature extraction and fusion strategies.

Summary updated

BibTeX
@article{pan2023multiscale,
  author = {Haodong Pan and Feng Gao and Junyu Dong and Qian Du},
  title = {{Multiscale Adaptive Fusion Network for Hyperspectral Image Denoising}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2023},
  doi = {10.1109/jstars.2023.3257051},
}
Figure