GAF-Net: Improving the Performance of Remote Sensing Image Fusion using Novel Global Self and Cross Attention Learning

Ankit Jha, Shirsha Bose, Biplab Banerjee

Introduction

Remote Sensing (RS) classification relies on the effective integration of multi-modal data from diverse sensors, such as Hyperspectral Imaging (HSI), LiDAR, and Synthetic Aperture Radar (SAR). While deep learning has significantly advanced fusion techniques, standard CNN-based methods often struggle with two primary issues: locality constraints and feature redundancy. CNNs typically rely on local convolutions, failing to capture the global context necessary for complex scene understanding. Additionally, joint training of multi-modal extractors often results in redundant features that do not fully exploit the unique information provided by each sensor.

To address these challenges, we present the Global Attention based Fusion Network (GAF-Net), a novel framework that utilizes global self-attention and auxiliary-task-driven cross-attention to achieve state-of-the-art performance in multi-modal classification.

The GAF-Net Architecture

The GAF-Net architecture is designed to capture long-range dependencies and reduce feature ambiguity. It consists of a main fusion branch ($S$) supported by two auxiliary branches ($T_1, T_2$) dedicated to individual modalities.

The overall architecture of GAF-Net, illustrating the integration of auxiliary tasks (T1, T2) and the main fusion branch (S) with MSFRB, SAB, and CAB modules.

Global Self-Attention Block (SAB)

The SAB is a core component that overcomes the locality constraints of traditional CNNs. Using a Query-Key-Value (QKV) approach, it captures global interactions across the entire image and all spectral channels simultaneously. The block is disentangled into three specific mechanisms:

  1. Global Spatial Attention (SA): Captures long-range spatial dependencies.
  2. Global Channel Attention (GCA): Models cross-channel correlations.
  3. Local Channel Attention (LCA): Captures fine-grained spectral variations.
Detailed breakdown of the Self-Attention Block (SAB) components: (a) Spatial Attention, (b) Global Channel Attention, (c) Local Channel Attention, and (d) the integrated SAB module.

Cross-Modal Attention and Regularization

Generating effective cross-modal masks is often difficult during joint training. GAF-Net solves this via the Cross-Attention Block (CAB), which utilizes high-level features from the auxiliary classification networks ($T_1$ and $T_2$). These "specialist" networks provide discriminative guidance to the main fusion branch, ensuring that the most relevant features from each modality are prioritized.

Furthermore, the authors introduce a Non-Redundancy Regularizer ($L_{NRR}$). This loss function penalizes high correlation between the attended features of different modalities, forcing the model to learn complementary information rather than redundant noise.

Experimental Results

GAF-Net was evaluated across five diverse benchmarks, including HSI-LiDAR (Houston 2013), HSI-SAR (Berlin & Augsburg), and even audio-visual data (ADVANCE).

  • State-of-the-Art Accuracy: On the Berlin HSI-SAR dataset, GAF-Net improved Overall Accuracy (OA) by approximately 8% over existing baselines like FusAtNet and CCR-Net.
  • Data Efficiency: Ablation studies demonstrated that the model maintains high performance even when training data is reduced to just 5% of the total set.
  • Cross-Domain Versatility: On the ADVANCE dataset, the model achieved a precision of 93.37%, a significant improvement over standard audio-visual models (~75%), proving its flexibility beyond traditional imagery.

Conclusion

GAF-Net effectively addresses the limitations of local context and feature redundancy in remote sensing fusion. By leveraging global attention and auxiliary task guidance, it provides a robust and versatile framework for multi-modal classification. Future research will explore extending this architecture to support N-modalities through sequential fusion and investigating more complex backbone architectures.

Summary updated

Abstract

The notion of self and cross-attention learning has been found to substantially boost the performance of remote sensing (RS) image fusion. However, while the self-attention models fail to incorporate the global context due to the limited size of the receptive fields, cross-attention learning may generate ambiguous features as the feature extractors for all the modalities are jointly trained. This results in the generation of redundant multi-modal features, thus limiting the fusion performance. To address these issues, we propose a novel fusion architecture called Global Attention based Fusion Network (GAF-Net), equipped with novel self and cross-attention learning techniques. We introduce the within-modality feature refinement module through global spectral-spatial attention learning using the query-key-value processing where both the global spatial and channel contexts are used to generate two channel attention masks. Since it is non-trivial to generate the cross-attention from within the fusion network, we propose to leverage two auxiliary tasks of modality-specific classification to produce highly discriminative cross-attention masks. Finally, to ensure non-redundancy, we propose to penalize the high correlation between attended modality-specific features. Our extensive experiments on five benchmark datasets, including optical, multispectral (MS), hyperspectral (HSI), light detection and ranging (LiDAR), synthetic aperture radar (SAR), and audio modalities establish the superiority of GAF-Net concerning the literature.
BibTeX
@inproceedings{jha2023gaf,
  author = {Ankit Jha and Shirsha Bose and Biplab Banerjee},
  title = {{GAF-Net: Improving the Performance of Remote Sensing Image Fusion using Novel Global Self and Cross Attention Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00629},
}
Figure