Abstract
Traditional change detection identifies spatial differences between images but lacks semantic context. This paper proposes Semantic Change Detection (SCD), a framework that classifies the nature of changes (e.g., "building destroyed" vs. "new car"). The core innovation, Hypermaps, extends the Hypercolumns technique by aggregating multi-scale convolutional features with Gaussian-weighted spatial context. Evaluated on the re-annotated TSUNAMI Panoramic dataset, the method achieves significant accuracy improvements over standard CNN-based baselines.
Introduction: From "Where" to "What"
Standard change detection is typically a binary task: identifying pixels that have changed between two timestamps. However, for applications like disaster response or urban planning, knowing where a change occurred is often insufficient. Stakeholders need to know what changed—distinguishing between a new construction project and disaster-induced rubble.
The primary challenge in automating this semantic understanding is the visual complexity of disaster scenes, which often feature textureless collapsed structures or visually similar classes like rubble and damaged buildings. Standard pixel-level classification often fails here because it lacks the necessary spatial context to distinguish between these subtle categories.
Methodology: The Hypermaps Framework
To address these challenges, the authors introduce Hypermaps, an evolution of the Hypercolumns approach.
Feature Aggregation
While standard Hypercolumns extract features at a single pixel across multiple CNN layers, Hypermaps aggregate values from a local neighborhood using a Gaussian-weighted sum. This approach captures:
- Spatial Detail: High-resolution features from early layers (e.g.,
pool2). - Semantic Context: High-level concepts from deeper layers (e.g.,
fc7). - Local Robustness: By blurring feature maps, the model smooths out noise and gains a better understanding of an object's immediate surroundings.
Multi-scale Representation
Objects in panoramic imagery vary significantly in scale. The system extracts features using three patch sizes (30x30, 50x50, and 70x70 pixels) and employs a "maximum count" voting mechanism to determine the final semantic label. The architecture utilizes a VGGNet backbone and classifies the resulting 4,736-dimensional feature vectors using Support Vector Machines (SVMs).
Experimental Setup
The study utilizes the TSUNAMI Panoramic Change Detection dataset, which contains 100 pairs of images. Since the original dataset only provided binary change masks, the authors re-annotated the changed areas with three semantic classes: Car (Blue), Building (Green), and Rubble (Red). This re-annotation enables the first rigorous benchmark for the SCD task in disaster scenarios.
Results and Discussion
The Multi-scale Weighted Hypermaps approach outperformed all baselines, including standard Hypercolumns and traditional hand-crafted features (HOG, SIFT+BoW).
Key performance metrics include:
- Accuracy: 71.18% at $t_0$ and 66.44% at $t_1$.
- Gain: A +4.64% ($t_0$) and +3.54% ($t_1$) improvement over standard Hypercolumns.
- Data Augmentation: Techniques like flipping and shifting were vital, boosting accuracy from 56.9% to over 67%.
Qualitatively, Hypermaps produce cleaner, more contiguous semantic regions compared to the noisy, "speckled" outputs of traditional pixel-wise CNN classification.
Conclusion
Semantic Change Detection represents a significant step forward in automated scene analysis. By leveraging Hypermaps to fuse multi-scale spatial and semantic information, the proposed framework provides the context necessary for complex decision-making in changing environments. Future work will focus on end-to-end training and expanding the taxonomy of detectable semantic classes.
Summary updated