Multi-Temporal Scene Classification and Scene Change Detection With Correlation Based Fusion

Lixiang Ru, Bo Du, Chen Wu

TL;DR

This paper introduces CorrFusion, a novel neural network module designed to improve how we classify land-use scenes and detect changes across multiple time periods. By explicitly modeling the temporal correlation between bi-temporal images (images of the same location at two different times), the authors achieve significant accuracy gains on a massive new benchmark, the Wuhan Dataset, which is orders of magnitude larger than previous scene-change datasets.

What Problem Does This Paper Address?

Traditional change detection in remote sensing often focuses on pixel-level or object-level changes (e.g., "this group of pixels turned from green to brown"). However, these don't always capture the semantic "big picture" of urban planning, such as a "Bare Land" area transitioning into a "Residential Region." This is known as scene-level change detection.

Existing deep learning methods for this task often treat the two images independently or simply concatenate their features, failing to exploit the inherent temporal correlation between them. Furthermore, the field has been held back by a lack of large-scale, diverse datasets, with most researchers relying on tiny datasets that lead to overfitting.

Methods

The authors propose CorrFusionNet, an end-to-end framework featuring the CorrFusion module. Here’s how it works:

  1. Feature Extraction: Two identical CNN branches (e.g., VGG, ResNet, or DenseNet) extract deep features from the bi-temporal images.
  2. Dimensionality Reduction & Decorrelation: Features are projected into a lower-dimensional space. The model uses Soft Deep Canonical Correlation Analysis (Soft DCCA) and a Soft Decorrelation Loss (SDL) to ensure the features are compact and non-redundant.
  3. Correlation-based Fusion: This is the "secret sauce." The module calculates the $L_2$ distance between the features of the two timepoints.
  4. If the scenes are highly similar (unchanged), a high weight is assigned, and features are fused via cross-temporal addition. This allows the network to "boost" the representation of one timepoint using information from the other.
  5. If the scenes are different (changed), the weight is low, preventing the features from interfering with each other.
  6. Joint Optimization: The network is trained using a multi-part loss function that combines standard classification error (Cross-Entropy) with decorrelation and correlation-specific losses.

Datasets

  • Wuhan Dataset (New): A major contribution of this paper. It contains 23,555 labeled scene image pairs (200x200 pixels) covering Wuhan, China, captured in 2014 and 2016. It features 14 categories, including "Administration," "Industrial," and various residential types.
  • Hanyang Dataset: A smaller, existing dataset with only 190 training pairs and 8 categories, used primarily for comparison.

Key Insights and Ideas

  • Similarity as a Filter: The core idea is that temporal correlation is a powerful prior. By using a $\tanh$-scaled weight based on feature distance, the model dynamically decides how much information to share between timepoints.
  • Moving Beyond Concatenation: Simple concatenation often forces a network to learn relationships from scratch. CorrFusion provides an explicit architectural "hint" that the two images are related in space and time.
  • The Importance of Scale: The results on the Wuhan dataset show that deeper models (like DenseNet) only truly shine when given enough data. On smaller datasets, simpler models or those with strong regularization are necessary to prevent overfitting.

Key Results

  • Accuracy Boost: On the Wuhan dataset, adding the CorrFusion module to a VGG16 backbone improved binary change detection accuracy (OA_bi) by approximately 6% and transition change detection (OA_tr) by about 5%.
  • State-of-the-Art Performance: CorrFusionNet outperformed existing methods like DCCANet and BoVW+KSFA across almost all metrics.
  • Backbone Agnostic: The module consistently improved performance regardless of whether the underlying "backbone" was VGG, ResNet, or DenseNet, proving its versatility.
  • Efficiency: The authors demonstrated that a dimensionality reduction ratio ($r$) of 2 to 8 maintains high accuracy while significantly reducing the number of parameters in the fusion layer.

Limitations and Future Work

  • Small Dataset Sensitivity: The authors noted that on the tiny Hanyang dataset, the improvements were less remarkable. This suggests the CorrFusion module requires a certain threshold of data to learn the correlation weights effectively.
  • Imbalanced Classes: The Wuhan dataset, like the real world, is imbalanced (e.g., far more "Water" and "Farmland" than "Playgrounds"). While the model performs well overall, accuracy on rare classes remains a challenge.
  • Generalization: While designed for bi-temporal remote sensing, the authors suggest the module could be adapted for other multi-view or multi-source classification problems, though this hasn't been tested yet.

Summary updated

Abstract

Classifying multi-temporal scene land-use categories and detecting their semantic scene-level changes for remote sensing imagery covering urban regions could straightly reflect the land-use transitions. Existing methods for scene change detection rarely focus on the temporal correlation of bi-temporal features, and are mainly evaluated on small scale scene change detection datasets. In this work, we proposed a CorrFusion module that fuses the highly correlated components in bi-temporal feature embeddings. We first extract the deep representations of the bi-temporal inputs with deep convolutional networks. Then the extracted features will be projected into a lower-dimensional space to extract the most correlated components and compute the instance-level correlation. The cross-temporal fusion will be performed based on the computed correlation in CorrFusion module. The final scene classification results are obtained with softmax layers. In the objective function, we introduced a new formulation to calculate the temporal correlation more efficiently and stably. The detailed derivation of backpropagation gradients for the proposed module is also given. Besides, we presented a much larger scale scene change detection dataset with more semantic categories and conducted extensive experiments on this dataset. The experimental results demonstrated that our proposed CorrFusion module could remarkably improve the multi-temporal scene classification and scene change detection results.
BibTeX
@article{ru2021multi,
  author = {Lixiang Ru and Bo Du and Chen Wu},
  title = {{Multi-Temporal Scene Classification and Scene Change Detection With Correlation Based Fusion}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=s9rC-I0RzrB},
}
Figure