Introduction
Remote Sensing (RS) classification relies on the effective integration of multi-modal data from diverse sensors, such as Hyperspectral Imaging (HSI), LiDAR, and Synthetic Aperture Radar (SAR). While deep learning has significantly advanced fusion techniques, standard CNN-based methods often struggle with two primary issues: locality constraints and feature redundancy. CNNs typically rely on local convolutions, failing to capture the global context necessary for complex scene understanding. Additionally, joint training of multi-modal extractors often results in redundant features that do not fully exploit the unique information provided by each sensor.
To address these challenges, we present the Global Attention based Fusion Network (GAF-Net), a novel framework that utilizes global self-attention and auxiliary-task-driven cross-attention to achieve state-of-the-art performance in multi-modal classification.
The GAF-Net Architecture
The GAF-Net architecture is designed to capture long-range dependencies and reduce feature ambiguity. It consists of a main fusion branch ($S$) supported by two auxiliary branches ($T_1, T_2$) dedicated to individual modalities.
Global Self-Attention Block (SAB)
The SAB is a core component that overcomes the locality constraints of traditional CNNs. Using a Query-Key-Value (QKV) approach, it captures global interactions across the entire image and all spectral channels simultaneously. The block is disentangled into three specific mechanisms:
- Global Spatial Attention (SA): Captures long-range spatial dependencies.
- Global Channel Attention (GCA): Models cross-channel correlations.
- Local Channel Attention (LCA): Captures fine-grained spectral variations.
Cross-Modal Attention and Regularization
Generating effective cross-modal masks is often difficult during joint training. GAF-Net solves this via the Cross-Attention Block (CAB), which utilizes high-level features from the auxiliary classification networks ($T_1$ and $T_2$). These "specialist" networks provide discriminative guidance to the main fusion branch, ensuring that the most relevant features from each modality are prioritized.
Furthermore, the authors introduce a Non-Redundancy Regularizer ($L_{NRR}$). This loss function penalizes high correlation between the attended features of different modalities, forcing the model to learn complementary information rather than redundant noise.
Experimental Results
GAF-Net was evaluated across five diverse benchmarks, including HSI-LiDAR (Houston 2013), HSI-SAR (Berlin & Augsburg), and even audio-visual data (ADVANCE).
- State-of-the-Art Accuracy: On the Berlin HSI-SAR dataset, GAF-Net improved Overall Accuracy (OA) by approximately 8% over existing baselines like FusAtNet and CCR-Net.
- Data Efficiency: Ablation studies demonstrated that the model maintains high performance even when training data is reduced to just 5% of the total set.
- Cross-Domain Versatility: On the ADVANCE dataset, the model achieved a precision of 93.37%, a significant improvement over standard audio-visual models (~75%), proving its flexibility beyond traditional imagery.
Conclusion
GAF-Net effectively addresses the limitations of local context and feature redundancy in remote sensing fusion. By leveraging global attention and auxiliary task guidance, it provides a robust and versatile framework for multi-modal classification. Future research will explore extending this architecture to support N-modalities through sequential fusion and investigating more complex backbone architectures.
Summary updated