TL;DR
This paper introduces LSAFNet, a novel late-stage fusion network for semantic change detection in remote sensing images. By decoupling semantic segmentation and change detection into separate branches that interact only at the decoder stage, LSAFNet achieves state-of-the-art performance while being more adaptable to pre-trained models.
What Problem Does This Paper Address?
Semantic Change Detection (SCD) is crucial for monitoring environmental changes using remote sensing data. Traditional approaches often struggle to accurately identify both the land cover categories and the specific changes between different time points. Existing multi-task learning methods typically entangle semantic segmentation (SS) and binary change detection (BCD) branches early in the network. This entanglement can hinder performance, make it difficult to leverage pre-trained models, and lead to insufficient refinement of bitemporal features, ultimately impacting accuracy.
Methods
LSAFNet proposes a more disentangled approach to SCD. It employs a multi-task learning framework with two parallel semantic segmentation (SS) branches and one binary change detection (BCD) branch. The key innovation is the late-stage fusion strategy, where the SS branches operate independently until the decoder stage, and their outputs are then fused for BCD.
To enhance feature representation, the authors introduce two novel modules:
- Local-Global Attentional Aggregation (LGAA) module: This module strengthens feature fusion by aggregating features from adjacent levels and incorporating both local and global context through a re-weighting mechanism. It's used in the Semantic Fusion Module (SFM) to create more representative features.
- Local-Global Context Enhancement (LGCE) module: This module is designed to highlight pivotal semantic information within the SS decoder by distinguishing important semantics from irrelevant ones, thereby improving feature refinement.
The BCD branch utilizes these refined features to explicitly model cross-temporal semantic differences. The overall loss function combines cross-entropy losses for SS and BCD, along with a semantic consistency loss to encourage alignment between the two tasks.
Datasets
The proposed LSAFNet is evaluated on two publicly available semantic change detection datasets:
- SECOND: Contains 2968 pairs of bitemporal images with resolutions ranging from 0.5m to 3.0m, covering categories like building, water, tree, low vegetation, ground, and playground.
- Landsat-SCD: Comprises 8468 pairs of bitemporal images with a 30m resolution, including categories like water, farmland, building, and desert.
Key Insights and Ideas
- Late-stage fusion is beneficial: Decoupling SS and BCD branches until the decoder stage allows for more focused feature learning within each task before they interact, leading to improved performance.
- Explicit feature refinement matters: Naive fusion strategies are insufficient. The LGAA and LGCE modules are crucial for refining bitemporal features by incorporating both local and global contextual information, which significantly boosts accuracy.
- Disentanglement aids foundation model adaptation: The proposed architecture's modularity and reduced entanglement make it easier to adapt pre-trained foundation models for SCD tasks, a significant advantage for future research.
- Multi-task learning synergy: While decoupled, the SS and BCD tasks still benefit from a joint training objective, with the semantic consistency loss playing a role in aligning their learning.
Key Results
LSAFNet achieves new state-of-the-art (SOTA) performance on both the SECOND and Landsat-SCD datasets. Specifically, on the SECOND dataset, it achieves an mIoU of 74.01% and an Fscd of 63.20%. On the Landsat-SCD dataset, it reaches an mIoU of 87.60% and an Fscd of 81.66%. Qualitative results demonstrate LSAFNet's ability to capture fine-grained changes and maintain high intra-category consistency. Ablation studies confirm the significant contributions of the LGAA and LGCE modules to the overall performance.
Limitations and Future Work
The authors acknowledge that their proposed architecture's disentanglement strengths can be further harnessed to adapt pre-trained foundation models into SCD fields in their future work. The paper focuses on improving the core architecture and its components, and future research will likely explore more advanced foundation model integration strategies.
Summary updated