A Late-Stage Bitemporal Feature Fusion Network for Semantic Change Detection

Chenyao Zhou, Haotian Zhang, Han Guo, Zhengxia Zou, Zhenwei ShiarXiv 2024 · OpenReview

TL;DR

This paper introduces LSAFNet, a novel late-stage fusion network for semantic change detection in remote sensing images. By decoupling semantic segmentation and change detection into separate branches that interact only at the decoder stage, LSAFNet achieves state-of-the-art performance while being more adaptable to pre-trained models.

What Problem Does This Paper Address?

Semantic Change Detection (SCD) is crucial for monitoring environmental changes using remote sensing data. Traditional approaches often struggle to accurately identify both the land cover categories and the specific changes between different time points. Existing multi-task learning methods typically entangle semantic segmentation (SS) and binary change detection (BCD) branches early in the network. This entanglement can hinder performance, make it difficult to leverage pre-trained models, and lead to insufficient refinement of bitemporal features, ultimately impacting accuracy.

Methods

LSAFNet proposes a more disentangled approach to SCD. It employs a multi-task learning framework with two parallel semantic segmentation (SS) branches and one binary change detection (BCD) branch. The key innovation is the late-stage fusion strategy, where the SS branches operate independently until the decoder stage, and their outputs are then fused for BCD.

To enhance feature representation, the authors introduce two novel modules:

  • Local-Global Attentional Aggregation (LGAA) module: This module strengthens feature fusion by aggregating features from adjacent levels and incorporating both local and global context through a re-weighting mechanism. It's used in the Semantic Fusion Module (SFM) to create more representative features.
  • Local-Global Context Enhancement (LGCE) module: This module is designed to highlight pivotal semantic information within the SS decoder by distinguishing important semantics from irrelevant ones, thereby improving feature refinement.

The BCD branch utilizes these refined features to explicitly model cross-temporal semantic differences. The overall loss function combines cross-entropy losses for SS and BCD, along with a semantic consistency loss to encourage alignment between the two tasks.

Datasets

The proposed LSAFNet is evaluated on two publicly available semantic change detection datasets:

  • SECOND: Contains 2968 pairs of bitemporal images with resolutions ranging from 0.5m to 3.0m, covering categories like building, water, tree, low vegetation, ground, and playground.
  • Landsat-SCD: Comprises 8468 pairs of bitemporal images with a 30m resolution, including categories like water, farmland, building, and desert.

Key Insights and Ideas

  • Late-stage fusion is beneficial: Decoupling SS and BCD branches until the decoder stage allows for more focused feature learning within each task before they interact, leading to improved performance.
  • Explicit feature refinement matters: Naive fusion strategies are insufficient. The LGAA and LGCE modules are crucial for refining bitemporal features by incorporating both local and global contextual information, which significantly boosts accuracy.
  • Disentanglement aids foundation model adaptation: The proposed architecture's modularity and reduced entanglement make it easier to adapt pre-trained foundation models for SCD tasks, a significant advantage for future research.
  • Multi-task learning synergy: While decoupled, the SS and BCD tasks still benefit from a joint training objective, with the semantic consistency loss playing a role in aligning their learning.

Key Results

LSAFNet achieves new state-of-the-art (SOTA) performance on both the SECOND and Landsat-SCD datasets. Specifically, on the SECOND dataset, it achieves an mIoU of 74.01% and an Fscd of 63.20%. On the Landsat-SCD dataset, it reaches an mIoU of 87.60% and an Fscd of 81.66%. Qualitative results demonstrate LSAFNet's ability to capture fine-grained changes and maintain high intra-category consistency. Ablation studies confirm the significant contributions of the LGAA and LGCE modules to the overall performance.

Limitations and Future Work

The authors acknowledge that their proposed architecture's disentanglement strengths can be further harnessed to adapt pre-trained foundation models into SCD fields in their future work. The paper focuses on improving the core architecture and its components, and future research will likely explore more advanced foundation model integration strategies.

Summary updated

Abstract

Semantic change detection is an important task in geoscience and earth observation. By producing a semantic change map for each temporal phase, both the land use land cover categories and change information can be interpreted. Recently some multi-task learning based semantic change detection methods have been proposed to decompose the task into semantic segmentation and binary change detection subtasks. However, previous works comprise triple branches in an entangled manner, which may not be optimal and hard to adopt foundation models. Besides, lacking explicit refinement of bitemporal features during fusion may cause low accuracy. In this letter, we propose a novel late-stage bitemporal feature fusion network to address the issue. Specifically, we propose local global attentional aggregation module to strengthen feature fusion, and propose local global context enhancement module to highlight pivotal semantics. Comprehensive experiments are conducted on two public datasets, including SECOND and Landsat-SCD. Quantitative and qualitative results show that our proposed model achieves new state-of-the-art performance on both datasets.
BibTeX
@misc{zhou2024late,
  author = {Chenyao Zhou and Haotian Zhang and Han Guo and Zhengxia Zou and Zhenwei Shi},
  title = {{A Late-Stage Bitemporal Feature Fusion Network for Semantic Change Detection}},
  year = {2024},
  eprint = {2406.10678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10678},
}
Figure