Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Lixiang Ru, Bo Du, Chen Wu

Introduction

In remote sensing, change detection has traditionally focused on pixel-level or object-level transitions. While useful, these methods often fail to capture the broader semantic shifts in urban landscapes—for instance, identifying when a "Bare Land" area becomes a "Residential Region." This higher-level task is known as scene-level change detection.

Comparison between pixel-level and scene-level change detection.

Existing deep learning approaches typically treat bi-temporal images (images of the same location at different times) independently or use simple feature concatenation. These methods often overlook the inherent temporal correlation between images, and the field has historically lacked large-scale datasets to train robust models.

The CorrFusionNet Framework

To address these challenges, the authors introduce CorrFusionNet, an end-to-end framework designed to explicitly model temporal correlations. The core of this framework is the CorrFusion module, which operates through a multi-stage process:

  1. Feature Extraction: Identical CNN branches (such as VGG or ResNet) extract deep features from the image pairs.
  2. Dimensionality Reduction & Decorrelation: Features are projected into a lower-dimensional space using Soft Deep Canonical Correlation Analysis (Soft DCCA) and a Soft Decorrelation Loss (SDL) to ensure compactness and reduce redundancy.
  3. Correlation-based Fusion: The module calculates the $L_2$ distance between temporal features. If scenes are similar (unchanged), a high weight is assigned to fuse features via cross-temporal addition, "boosting" the representation. If scenes have changed, the weight is low to prevent feature interference.
  4. Joint Optimization: The network is trained with a composite loss function combining Cross-Entropy with decorrelation and correlation-specific losses.
Detailed architecture of the proposed CorrFusion module.

Data and Benchmarking

A significant contribution of this work is the introduction of the Wuhan Dataset. Comprising 23,555 labeled scene image pairs from 2014 and 2016, it is orders of magnitude larger than previous benchmarks like the Hanyang dataset (190 pairs). This scale allows deeper architectures like DenseNet to be effectively trained without immediate overfitting.

Key Results and Insights

The CorrFusion module demonstrates significant performance gains:

  • Accuracy Improvements: On the Wuhan dataset, adding CorrFusion to a VGG16 backbone increased binary change detection accuracy (OA_bi) by approximately 6% and transition change detection (OA_tr) by 5%.
  • Versatility: The module is "backbone agnostic," consistently improving results across VGG, ResNet, and DenseNet architectures.
  • Efficiency: By optimizing the dimensionality reduction ratio ($r$), the model maintains high accuracy while significantly reducing parameter counts in the fusion layer.

Conclusion

CorrFusionNet represents a shift toward more semantically aware change detection. By treating temporal similarity as a filter rather than just data to be concatenated, the model more effectively captures the dynamics of land-use change. Future work will likely focus on addressing class imbalances and extending this correlation-based approach to multi-source data fusion.

Summary updated

BibTeX
@article{ru2021multi,
  author = {Lixiang Ru and Bo Du and Chen Wu},
  title = {{Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=s9rC-I0RzrB},
}
Figure