Explicit Change Relation Learning for Change Detection in VHR Remote Sensing Images

Dalong Zheng, Zebin Wu, Jia Liu, Chih-Cheng Hung, Zhihui WeiIEEE GRSL 2023 · OpenReview

TL;DR

This paper introduces NAME, a novel triple-branch network for change detection in very high-resolution (VHR) remote sensing images. It explicitly models pre-change, post-change, and crucially, the relationship between these changes, leading to improved detection accuracy. A key innovation is the Continuous Change Relation (CCR) branch, which captures fine-grained temporal information to enhance change discrimination.

What Problem Does This Paper Address?

Interpreting remote sensing images often involves identifying changes over time, a task known as change detection. This is fundamentally a binary classification problem: is there a change at this location or not? A major challenge is distinguishing "semantic changes" (e.g., new buildings, deforestation) from "non-semantic changes" (e.g., variations in lighting, shadows, seasonal foliage changes). Existing deep learning methods, while powerful, often implicitly learn these relationships within their architectures. The authors argue that this implicit learning is insufficient, leading to a loss of crucial semantic change information and ultimately poorer performance. They propose that explicitly modeling three distinct types of features – pre-change, post-change, and the relationship between them – is key to overcoming this limitation.

Methods

The core of the proposed NAME architecture is a triple-branch network. Each branch is designed to extract and process a specific type of feature:

  • CNN Branch & Transformer Branch: These two parallel branches work together to extract local (CNN) and global (Transformer) features from each input image. This hybrid approach aims to capture both detailed textures and broader contextual information.
  • Structured Change Relation (SCR) Branch: This branch is a serial combination of Transformer (Swin-V2) and CNN blocks. It's designed to explicitly mine the "structured and continuous change relationship features" by interacting and fusing the features from the CNN and Transformer branches.
  • Continuous Change Relation (CCR) Branch: This novel branch addresses the limitations of implicit temporal modeling. Instead of relying on complex temporal encoders, it constructs "pseudo video frames" (PVFs) through linear interpolation between the two input images. These PVFs are then processed by CCR blocks to extract detailed, continuous change relationship features. This branch is designed to be a "plug-and-play" module that can also benefit other change detection networks.

The features from these branches are then fused in the decoder stage, with additional detail features from the SCR and CCR branches complementing the process. Finally, an ensemble approach combines the outputs of multiple decoders to produce the final change map.

Datasets

The authors evaluated their NAME model on four widely used Very High Resolution (VHR) remote sensing datasets:

  • LEVIR-CD: Learning, Vision, and Remote Sensing Change Detection Dataset.
  • SVCD: Season-Varying Change Detection Dataset.
  • WHU-CD: Wuhan University Change Detection Dataset.
  • SYSU-CD: Sun Yat-sen University Change Detection Dataset.

These datasets feature images with very high spatial resolutions, ranging from 0.03 to 1 meter per pixel.

Key Insights and Ideas

  • Explicitly Modeling Change Relationships is Crucial: The paper strongly advocates for treating "change relationship features" as a distinct and important category, rather than implicitly learning them. This explicit modeling is presented as the key to improved semantic change detection.
  • Triple-Branch Architecture for Comprehensive Feature Extraction: Separating features into pre-change, post-change, and change-relation allows for a more targeted and effective extraction and fusion process.
  • The Power of Pseudo Video Frames (PVFs) for Fine-Grained Temporal Information: The CCR branch's use of PVFs, derived from linear interpolation, offers a novel way to capture continuous and detailed change information that might be missed by other methods. This is particularly effective for VHR images where subtle changes are important.
  • CCR Branch as a Versatile Module: The authors demonstrate that the CCR branch is not only effective within their NAME architecture but can also be integrated into other change detection networks to boost their performance, highlighting its general utility.

Key Results

The NAME model consistently outperformed existing advanced change detection methods across the four VHR datasets. Specifically, the paper reports the following F1 score improvements over the second-best method:

  • LEVIR-CD: 0.66%
  • SVCD: 0.37%
  • WHU-CD: 0.70%
  • SYSU-CD: 1.09%

The authors also highlight that their model exhibits better robustness and more accurate detection capabilities, particularly for structured and small objects, as evidenced by comprehensive indicators like F1 and Overall Accuracy (OA).

Limitations and Future Work

  • Computational Complexity: While not explicitly detailed as a limitation, the use of multiple branches and transformer components suggests a potentially higher computational cost compared to simpler models.
  • Ablation Study on CCR Frames: The paper includes an ablation study on the number of frames used in the CCR branch, showing that increasing frames generally improves performance, with 4 frames yielding the best results. This suggests that further exploration into optimal PVF construction could be beneficial.
  • Future Work: The authors express hope that their work will inspire new approaches to mining change relationships in change detection. The plug-and-play nature of the CCR branch suggests its potential integration into a wider range of change detection frameworks.

Summary updated

Abstract

Change detection has always been a concerned task in the interpretation of remote sensing images. It is essentially a unique binary classification task with two inputs, and there is a change relationship between these two inputs. At present, the mining of change relationship features is usually implicit in the network architectures that contain single-branch or two-branch encoders. However, due to the lack of artificial prior design for change relationship features, these networks cannot learn enough change semantic information and lose more accurate change detection performance. So we propose a network architecture NAME for the explicit mining of change relation features. In our opinion, the change features of change detection should be divided into pre-changed image features, post-changed image features and change relation features. In order to fully mine these three kinds of change features, we propose the triple branch network combining the transformer and convolutional neural network (CNN) to extract and fuse these change features from two perspectives of global information and local information, respectively. In addition, we design the continuous change relation (CCR) branch to further obtain the continuous and detail change relation features to improve the change discrimination capability of the model. The experimental results show that our network performs better, in terms of F1, IoU, and OA, than those of the existing advanced networks for change detection on four public very high-resolution (VHR) remote sensing datasets. Our source code is available at https://github.com/DalongZ/NAME.
BibTeX
@article{zheng2023explicit,
  author = {Dalong Zheng and Zebin Wu and Jia Liu and Chih-Cheng Hung and Zhihui Wei},
  title = {{Explicit Change Relation Learning for Change Detection in VHR Remote Sensing Images}},
  journal = {IEEE Geoscience and Remote Sensing Letters},
  year = {2023},
  url = {https://openreview.net/forum?id=T59OcqArJ6},
}
Figure