HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images

Chengxi Han, Chen Wu, Haonan Guo, Meiqi Hu, Hongruixuan Chen

TL;DR

This paper introduces HANet, a novel hierarchical attention network for change detection in very-high-resolution remote sensing images. To address the significant class imbalance in these datasets, they propose a Progressive Foreground-Balanced Sampling (PFBS) strategy and a discriminative Siamese network with a lightweight attention module. Their method achieves state-of-the-art performance on challenging benchmarks.

What Problem Does This Paper Address?

Change detection (CD) in remote sensing (RS) images aims to identify differences in an object or phenomenon observed at different times. Very-high-resolution (VHR) RS imagery is particularly valuable for applications like land use analysis and urban monitoring. However, a major challenge in this domain is the extreme imbalance between changed and unchanged pixels. Typically, changed pixels constitute a very small fraction of the total image, making it difficult for deep learning models to learn the features of changed areas effectively. Existing methods often struggle with this imbalance, leading to missed detections or inaccurate results. Furthermore, VHR images present complex spatial details and variations due to illumination, seasonal changes, and environmental factors, which traditional methods and even some deep learning approaches find difficult to handle.

Methods

The authors propose two main innovations:

  • Progressive Foreground-Balanced Sampling (PFBS): This strategy aims to mitigate the class imbalance problem without adding computational overhead or modifying the change information itself. The core idea is to prioritize learning from the minority "changed" (foreground) pixels during the early stages of training. PFBS employs different training policies, such as "Fixed-X" (training only on foreground images for X epochs) and "Linear-Y" (gradually introducing background images over Y epochs), or a combination of both. This progressive approach helps the model learn crucial change features before being overwhelmed by the majority of unchanged pixels.
  • Hierarchical Attention Network (HANet): This is a discriminative Siamese network designed to integrate multi-scale features and refine detailed spatial and temporal information.
  • Multi-scale Feature Extractor: HANet extracts features at four different scales from the input bi-temporal VHR images.
  • Hierarchical Attention (HAN) Module: This module is the core of HANet and is designed to be lightweight and effective. It combines a Parallel Convolutional Structure (PCS) for multi-scale feature fusion with spatial-spectral axial attention (Column and Row Attention - Col.-Row-A). PCS uses group convolutions with different dilation rates to capture multi-scale information efficiently. Col.-Row-A, inspired by attention mechanisms in natural language processing, refines features by considering relationships across both spatial and spectral dimensions, while being computationally more efficient than standard self-attention.

Datasets

The proposed method was evaluated on two widely used, extremely unbalanced VHR remote sensing image datasets:

  • WHU-CD: A public dataset focused on building change detection, featuring areas in New Zealand affected by an earthquake. It has a very low percentage of changed pixels (4.26%).
  • LEVIR-CD: A large-scale dataset containing VHR image pairs with significant seasonal and illumination variations, making change detection more challenging. It also exhibits a low percentage of changed pixels (4.65%).

The authors also mention using a subset of the WHU-CD training data for validation.

Key Insights and Ideas

  • Class imbalance is a critical bottleneck: The paper strongly emphasizes that the extreme imbalance between changed and unchanged pixels is a primary reason for the limitations of existing CD methods, even advanced deep learning ones.
  • Prioritizing minority class learning is effective: The PFBS strategy demonstrates that strategically focusing on the minority class (changed pixels) early in training can significantly improve a model's ability to learn relevant features without altering the data itself.
  • Hierarchical attention captures rich details: The HAN module, by combining multi-scale feature extraction with efficient spatial-spectral attention, proves adept at capturing both broad contextual information and fine-grained details of buildings, leading to more accurate change detection.
  • Lightweight attention is crucial for VHR data: The design of the HAN module as a lightweight attention mechanism is important for handling the high dimensionality and complexity of VHR imagery without incurring excessive computational costs.

Key Results

  • State-of-the-art performance: HANet achieved superior performance compared to several state-of-the-art methods on both WHU-CD and LEVIR-CD datasets, particularly on key metrics like F1-score, Overall Accuracy (OA), and Kappa Coefficient (KC). For instance, on WHU-CD, HANet achieved an F1-score of 88.16%, outperforming the next best method (RSP-BIT) by 0.66%.
  • Improved handling of false negatives/positives: Qualitative results (Fig. 6) show that HANet is better at avoiding missed detections (false negatives, blue pixels) and erroneous detections (false positives, red pixels) on complex building features and backgrounds.
  • Effective PFBS: Ablation studies confirmed the effectiveness of PFBS, showing that even simple strategies like Fixed-X (e.g., Fixed-15) significantly improved performance over the baseline. The combination of Fixed-X and Linear-Y further enhanced results.
  • Efficient attention: The HAN module, particularly the Col.-Row-A component, contributes to improved performance while maintaining a relatively low parameter count (3.03M) and FLOPs compared to other attention-based or transformer-based methods.

Limitations and Future Work

  • Hyperparameter tuning for PFBS: The authors acknowledge that finding the optimal values for X and Y in the PFBS strategy (Fixed-X and Linear-Y) might require further investigation.
  • Exploring more PFBS variations: They suggest exploring more diverse forms of PFBS, such as nonlinear increases or combinations with other sampling techniques.
  • Generalizability of PFBS: The paper proposes investigating the application of PFBS with other CD models and on additional datasets to further validate its effectiveness.
  • HAN module migration: The HAN module is suggested as a potentially transferable component for improving performance in other remote sensing tasks.

Summary updated

Abstract

Benefiting from the developments in deep learning technology, deep learning-based algorithms employing automatic feature extraction have achieved remarkable performance on the change detection (CD) task. However, the performance of existing deep learning-based CD methods is hindered by the imbalance between changed and unchanged pixels. To tackle this problem, a progressive foreground-balanced sampling (PFBS) strategy on the basis of not adding change information is proposed to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance. Furthermore, we design a discriminative Siamese network, Hierarchical Attention Network (HANet), which can integrate multi-scale features and refine detailed features. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism. Extensive experiments and ablation studies on two CD datasets with extremely unbalanced labels validate the effectiveness and efficiency of the proposed method. Our model is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ChengxiHAN/HANet-CD</uri> .
BibTeX
@article{han2023hanet,
  author = {Chengxi Han and Chen Wu and Haonan Guo and Meiqi Hu and Hongruixuan Chen},
  title = {{HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images}},
  journal = {IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
  year = {2023},
  doi = {10.1109/jstars.2023.3264802},
}
Figure