Hybrid Attention Fusion Embedded in Transformer for Remote Sensing Image Semantic Segmentation

Yan Chen, Quan Dong, Xiaofeng Wang, Qianchuan Zhang, Menglei Kang, Wenxiang Jiang, Mengyuan Wang, Lixiang Xu, Chen ZhangIEEE JSTARS 2024 · OpenReview

Abstract

In the context of fast progress in deep learning, convolutional neural networks have been extensively applied to the semantic segmentation of remote sensing images and have achieved significant progress. However, certain limitations exist in capturing global contextual information due to the characteristics of convolutional local properties. Recently, Transformer has become a focus of research in computer vision and has shown great potential in extracting global contextual information, further promoting the development of semantic segmentation tasks. In this article, we use ResNet50 as an encoder, embed the hybrid attention mechanism into Transformer, and propose a Transformer-based decoder. The Channel-Spatial Transformer Block further aggregates features by integrating the local feature maps extracted by the encoder with their associated global dependencies. At the same time, an adaptive approach is employed to reweight the interdependent channel maps to enhance the feature fusion. The global cross-fusion module combines the extracted complementary features to obtain more comprehensive semantic information. Extensive comparative experiments were conducted on the ISPRS Potsdam and Vaihingen datasets, where mIoU reached 78.06% and 76.37%, respectively. The outcomes of multiple ablation experiments also validate the effectiveness of the proposed method.

Introduction

Semantic segmentation in remote sensing—assigning a category label to every pixel in an aerial or satellite image—is a critical yet challenging task. Unlike natural images, remote sensing data often exhibits high intra-class variance, where objects of the same category (like buildings) appear different due to materials or shadows. Conversely, it also suffers from low inter-class variance, where distinct categories (like trees and low vegetation) share similar spectral and textural signatures. While standard Convolutional Neural Networks (CNNs) are efficient at capturing local details, their limited receptive fields hinder their ability to model global contextual relationships. Transformers offer a solution for global reasoning but can be computationally heavy and may lose fine-grained spatial information when used as standalone encoders.

Methodology

The authors propose HAFNet, a hybrid architecture that leverages a ResNet50 encoder for efficient local feature extraction and a novel Transformer-based decoder for global context integration. The architecture is defined by two key modules:

Channel-Spatial Transformer Block (CSTB)

The CSTB replaces standard self-attention with a dual-branch mechanism. It incorporates a Channel Adaptive Module (CAM) to dynamically reweight feature channels and a Spatial Context Module (SCM) to capture long-range spatial dependencies. Additionally, the block utilizes a modified Multi-Layer Perceptron (MLP) with 7x7 depth-wise convolutions and Global Response Normalization (GRN) to maintain high feature contrast and prevent model collapse.

Global Cross-Fusion Module (GCFM)

To ensure high-fidelity upsampling, the GCFM employs an interactive fusion strategy. It correlates high-level semantic information with low-level spatial features from the encoder's skip connections, preserving sharp boundaries for small or complex objects like cars and building edges.

Experimental Setup

HAFNet was evaluated on the ISPRS Potsdam and ISPRS Vaihingen datasets. These benchmarks consist of high-resolution aerial imagery categorized into six classes: impervious surfaces, buildings, low vegetation, trees, cars, and clutter. The researchers utilized standard data augmentation techniques and trained the model on 512x512 image patches.

Results and Analysis

HAFNet achieved state-of-the-art performance, reaching a mIoU of 78.06% on Potsdam and 76.37% on Vaihingen. A notable achievement is the model's performance on the "Clutter" category, where it outperformed existing networks by over 4.63% in IoU, demonstrating its robustness in identifying irregular and complex structures.

Qualitative comparison of HAFNet against other models on the Potsdam and Vaihingen datasets.

Qualitative analysis (as seen in the figure above) shows that HAFNet produces significantly cleaner segmentation maps with smoother building boundaries and fewer misclassifications between trees and low vegetation compared to baseline models like MANet and MsanlfNet.

Conclusion

By embedding a hybrid attention mechanism within a Transformer-based decoder, HAFNet effectively bridges the gap between local feature extraction and global contextual reasoning. While the model is more computationally intensive than some lightweight alternatives, its superior accuracy on complex urban datasets marks a significant advancement in remote sensing segmentation. Future research will focus on model optimization for real-time applications and further integration of CNN-Transformer synergies.

Summary updated

Abstract

In the context of fast progress in deep learning, convolutional neural networks have been extensively applied to the semantic segmentation of remote sensing images and have achieved significant progress. However, certain limitations exist in capturing global contextual information due to the characteristics of convolutional local properties. Recently, Transformer has become a focus of research in computer vision and has shown great potential in extracting global contextual information, further promoting the development of semantic segmentation tasks. In this article, we use ResNet50 as an encoder, embed the hybrid attention mechanism into Transformer, and propose a Transformer-based decoder. The Channel-Spatial Transformer Block further aggregates features by integrating the local feature maps extracted by the encoder with their associated global dependencies. At the same time, an adaptive approach is employed to reweight the interdependent channel maps to enhance the feature fusion. The global cross-fusion module combines the extracted complementary features to obtain more comprehensive semantic information. Extensive comparative experiments were conducted on the ISPRS Potsdam and Vaihingen datasets, where mIoU reached 78.06% and 76.37%, respectively. The outcomes of multiple ablation experiments also validate the effectiveness of the proposed method.
BibTeX
@article{chen2024hybrid,
  author = {Yan Chen and Quan Dong and Xiaofeng Wang and Qianchuan Zhang and Menglei Kang and Wenxiang Jiang and Mengyuan Wang and Lixiang Xu and Chen Zhang},
  title = {{Hybrid Attention Fusion Embedded in Transformer for Remote Sensing Image Semantic Segmentation}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=QgWyCMGEwV},
}
Figure