Semantic Change Detection with Hypermaps

Hirokatsu Kataoka, Soma Shirakabe, Yudai Miyashita, Akio Nakamura, Kenji Iwata, Yutaka SatoharXiv 2016 · arXiv

Abstract

Traditional change detection identifies spatial differences between images but lacks semantic context. This paper proposes Semantic Change Detection (SCD), a framework that classifies the nature of changes (e.g., "building destroyed" vs. "new car"). The core innovation, Hypermaps, extends the Hypercolumns technique by aggregating multi-scale convolutional features with Gaussian-weighted spatial context. Evaluated on the re-annotated TSUNAMI Panoramic dataset, the method achieves significant accuracy improvements over standard CNN-based baselines.

Introduction: From "Where" to "What"

Standard change detection is typically a binary task: identifying pixels that have changed between two timestamps. However, for applications like disaster response or urban planning, knowing where a change occurred is often insufficient. Stakeholders need to know what changed—distinguishing between a new construction project and disaster-induced rubble.

Semantic change detection aims to identify specific events like new construction or structural damage.

The primary challenge in automating this semantic understanding is the visual complexity of disaster scenes, which often feature textureless collapsed structures or visually similar classes like rubble and damaged buildings. Standard pixel-level classification often fails here because it lacks the necessary spatial context to distinguish between these subtle categories.

Methodology: The Hypermaps Framework

To address these challenges, the authors introduce Hypermaps, an evolution of the Hypercolumns approach.

Feature Aggregation

While standard Hypercolumns extract features at a single pixel across multiple CNN layers, Hypermaps aggregate values from a local neighborhood using a Gaussian-weighted sum. This approach captures:

  1. Spatial Detail: High-resolution features from early layers (e.g., pool2).
  2. Semantic Context: High-level concepts from deeper layers (e.g., fc7).
  3. Local Robustness: By blurring feature maps, the model smooths out noise and gains a better understanding of an object's immediate surroundings.

Multi-scale Representation

Objects in panoramic imagery vary significantly in scale. The system extracts features using three patch sizes (30x30, 50x50, and 70x70 pixels) and employs a "maximum count" voting mechanism to determine the final semantic label. The architecture utilizes a VGGNet backbone and classifies the resulting 4,736-dimensional feature vectors using Support Vector Machines (SVMs).

Experimental Setup

The study utilizes the TSUNAMI Panoramic Change Detection dataset, which contains 100 pairs of images. Since the original dataset only provided binary change masks, the authors re-annotated the changed areas with three semantic classes: Car (Blue), Building (Green), and Rubble (Red). This re-annotation enables the first rigorous benchmark for the SCD task in disaster scenarios.

Results and Discussion

The Multi-scale Weighted Hypermaps approach outperformed all baselines, including standard Hypercolumns and traditional hand-crafted features (HOG, SIFT+BoW).

The proposed method generates semantic maps for pre-disaster (t0) and post-disaster (t1) states, enabling automated event identification.

Key performance metrics include:

  • Accuracy: 71.18% at $t_0$ and 66.44% at $t_1$.
  • Gain: A +4.64% ($t_0$) and +3.54% ($t_1$) improvement over standard Hypercolumns.
  • Data Augmentation: Techniques like flipping and shifting were vital, boosting accuracy from 56.9% to over 67%.

Qualitatively, Hypermaps produce cleaner, more contiguous semantic regions compared to the noisy, "speckled" outputs of traditional pixel-wise CNN classification.

Conclusion

Semantic Change Detection represents a significant step forward in automated scene analysis. By leveraging Hypermaps to fuse multi-scale spatial and semantic information, the proposed framework provides the context necessary for complex decision-making in changing environments. Future work will focus on end-to-end training and expanding the taxonomy of detectable semantic classes.

Summary updated

Abstract

Change detection is the study of detecting changes between two different images of a scene taken at different times. By the detected change areas, however, a human cannot understand how different the two images. Therefore, a semantic understanding is required in the change detection research such as disaster investigation. The paper proposes the concept of semantic change detection, which involves intuitively inserting semantic meaning into detected change areas. We mainly focus on the novel semantic segmentation in addition to a conventional change detection approach. In order to solve this problem and obtain a high-level of performance, we propose an improvement to the hypercolumns representation, hereafter known as hypermaps, which effectively uses convolutional maps obtained from convolutional neural networks (CNNs). We also employ multi-scale feature representation captured by different image patches. We applied our method to the TSUNAMI Panoramic Change Detection dataset, and re-annotated the changed areas of the dataset via semantic classes. The results show that our multi-scale hypermaps provided outstanding performance on the re-annotated TSUNAMI dataset.
BibTeX
@misc{kataoka2016semantic,
  author = {Hirokatsu Kataoka and Soma Shirakabe and Yudai Miyashita and Akio Nakamura and Kenji Iwata and Yutaka Satoh},
  title = {{Semantic Change Detection with Hypermaps}},
  year = {2016},
  eprint = {1604.07513},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.07513},
}
Figure