Dsfer-Net: A Deep Supervision and Feature Retrieval Network for Bitemporal Change Detection Using Modern Hopfield Networks

Shizhen Chang, Michael Kopp, Pedram GhamisiIEEE TGRS 2023 · DOI · OpenReview

TL;DR

This paper introduces Dsfer-Net, a novel deep learning architecture for detecting changes in remote sensing images. It leverages modern Hopfield networks for enhanced feature retrieval and deep supervision to improve the accuracy and interpretability of change detection, outperforming existing state-of-the-art methods on several benchmark datasets.

What Problem Does This Paper Address?

Change detection in remote sensing images is crucial for monitoring land surface evolution, with applications ranging from urban planning to disaster assessment. While deep learning methods have shown promise, they often struggle to effectively capture and interpret subtle changes, especially in high-resolution imagery with complex textures. Existing approaches often lack clear explanations for how deep features contribute to detection performance and can be prone to false alarms due to factors like illumination changes or shadows. This paper aims to address these limitations by developing a more robust and explainable change detection network.

Methods

The core of Dsfer-Net is a Siamese convolutional neural network that extracts multi-level deep features from bitemporal images. The novelty lies in the integration of a Deeply Supervised Feature Retrieval (DSFR) module and a Comprehensive Fusion (CF) strategy.

  • DSFR Module: This module, inspired by modern Hopfield networks (MHNs), is designed to aggregate and retrieve difference features from deeper network layers. MHNs are used because they are differentiable and can learn from continuous inputs, allowing for gradient-based training. The DSFR module aims to capture semantic concepts and spatial structures, providing a more discriminative representation of changes. Deep supervision is applied to ensure the comprehensive restoration and retrieval of these features.
  • Comprehensive Fusion (CF) Strategy: This strategy is used in the decoder to fuse multi-scale features. It incorporates the retrieved features from the DSFR module as a "weight mask" to emphasize changed regions and suppress unchanged ones. This fusion process is designed to be more effective than simple concatenation by leveraging the learned change information.
  • Hybrid Loss Function: The network is trained using a combination of weighted binary cross-entropy loss (to handle class imbalance) and dice loss (for feature retrieval).

Datasets

The proposed Dsfer-Net was evaluated on three widely used benchmark datasets for bitemporal change detection:

  • LEVIR-CD: A large-scale building change detection dataset with very high-resolution Google Earth imagery.
  • WHU-CD: A building change detection dataset focusing on an area in New Zealand affected by an earthquake, capturing pre- and post-reconstruction imagery.
  • CDD: A dataset designed to capture seasonal variations and object changes, including images from Google Earth.

Key Insights and Ideas

  • Modern Hopfield Networks for Feature Retrieval: The paper demonstrates that MHNs can be effectively adapted for change detection by using them to retrieve and aggregate difference features from deep layers. This goes beyond traditional feature extraction by actively seeking out and emphasizing changes.
  • Deep Supervision for Explainability: The deep supervision applied to the DSFR module not only improves performance but also offers a degree of explainability, showing how the network learns to identify changes at different levels of abstraction.
  • Feature Retrieval as a Mask: Using the retrieved deep change features as a mask in the fusion process is a key innovation. This allows the network to intelligently combine information from different scales, focusing on areas where changes are most likely.
  • Mid-Level Features are Crucial: Visualizations suggest that change information is often optimally preserved in mid-level features, and the DSFR module effectively leverages this by operating on deeper layers.

Key Results

Dsfer-Net achieved state-of-the-art performance on all three benchmark datasets.

  • LEVIR-CD: F1 scores increased by 4.7% compared to DSAMNet and 2.0% compared to FrNet.
  • WHU-CD: F1 scores increased by 5.9% compared to DSAMNet and 1.4% compared to FrNet.
  • CDD: F1 scores increased by 2.3% compared to DSAMNet and 4.5% compared to FrNet.

Qualitatively, Dsfer-Net demonstrated superior performance in maintaining the shape and preserving the boundaries of changed objects, producing clearer contours and fewer false alarms and missed detections compared to other methods.

Limitations and Future Work

  • Parameter Count: The paper acknowledges that the use of Hopfield layers increases the parameter count of the model, although it maintains competitive inference times.
  • Sensitivity to Pseudo-Changes: The authors note that in scenes with many unexpected changes, the feature retrieval module might struggle to accurately locate changes in deeper layers.
  • Future Work: The authors plan to investigate the impact of different training ratios on detection accuracy and explore further applications of the Hopfield layer for semantic segmentation. They also aim to design structures to overcome the side effects of pseudo-changes.

Summary updated

Abstract

Change detection, as an important application for high-resolution remote sensing images, aims to monitor and analyze changes in the land surface over time. With the rapid growth in the quantity of high-resolution remote sensing data and the complexity of texture features, a number of quantitative deep learning-based methods have been proposed. Although these methods outperform traditional change detection methods by extracting deep features and combining spatial-temporal information, reasonable explanations about how deep features work on improving the detection performance are still lacking. In our investigations, we find that modern Hopfield network layers achieve considerable performance in semantic understandings. In this paper, we propose a Deep Supervision and FEature Retrieval network (Dsfer-Net) for bitemporal change detection. Specifically, the highly representative deep features of bitemporal images are jointly extracted through a fully convolutional Siamese network. Based on the sequential geo-information of the bitemporal images, we then design a feature retrieval module to retrieve the difference feature and leverage discriminative information in a deeply supervised manner. We also note that the deeply supervised feature retrieval module gives explainable proofs about the semantic understandings of the proposed network in its deep layers. Finally, this end-to-end network achieves a novel framework by aggregating the retrieved features and feature pairs from different layers. Experiments conducted on three public datasets (LEVIR-CD, WHU-CD, and CDD) confirm the superiority of the proposed Dsfer-Net over other state-of-the-art methods. Code will be available online (https://github.com/ShizhenChang/Dsfer-Net).
BibTeX
@article{chang2023dsfer,
  author = {Shizhen Chang and Michael Kopp and Pedram Ghamisi},
  title = {{Dsfer-Net: A Deep Supervision and Feature Retrieval Network for Bitemporal Change Detection Using Modern Hopfield Networks}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  doi = {10.1109/tgrs.2024.3424532},
}
Figure