MaskSCD: Bitemporal Mask Classification for Semantic Change Detection in Remote Sensing Images

Weikang Yu, Pedram GhamisiarXiv 2026 · arXiv

In the world of remote sensing, Semantic Change Detection (SCD) is the "holy grail." It doesn't just tell you that something changed (binary change detection); it tells you what changed into what (e.g., "farmland to buildings").

However, traditional deep learning models for SCD suffer from a nagging logical flaw: they often predict change maps and land-cover maps that contradict each other. A new paper, MaskSCD, proposes a fundamental shift in how we frame this problem, moving from per-pixel classification to a more holistic "mask classification" approach.

TL;DR

MaskSCD formulates semantic change detection as the mask classification of a bitemporal label-pair image. By predicting a single mask shared by both dates and a class label for each date, it ensures 100% consistency between change and semantic maps while achieving state-of-the-art accuracy across major benchmarks.

What Problem Does This Paper Address?

Most current SCD networks use a "per-pixel multi-task" design. They employ a Siamese encoder to extract features from two images ($I^1$ and $I^2$) and then use three separate decoders: one for a binary change map and two for semantic maps at each date.

This leads to two major issues:

  1. Contradictory Pixels: Because the decoders are independent, a model might label a pixel as "changed" even if it assigns it the same class (e.g., "water") at both dates. In some models, these contradictions account for nearly 30% of the predicted change.
  2. Object Fragmentation: Per-pixel decisions often result in "salt-and-pepper" noise, jagged boundaries, and holes within otherwise homogeneous objects like buildings or fields.
Comparison of traditional per-pixel multi-task SCD (a) vs. the proposed MaskSCD (b). Note the magenta "contradiction" pixels in the traditional approach.

Methods

The authors introduce MaskSCD, which treats SCD as the segmentation of a bitemporal label-pair image. Instead of classifying every pixel, the model predicts a set of "bitemporal segments"—regions where the pair of land-cover classes remains constant.

Key Innovations:

  • Bitemporal Mask Classification: Each query in the transformer decoder predicts one mask ($M_i$) shared by both dates and two class distributions ($p^1_i, p^2_i$). The change map is then simply derived: a pixel is "changed" if $p^1_i \neq p^2_i$. This makes contradictions mathematically impossible.
  • Bitemporal Fusion Pyramid (BFP): This module fuses features from both dates at multiple scales using bidirectional cross-temporal attention. It keeps per-date features separate while allowing them to "talk" to each other to identify change-aware signals.
  • Bitemporal Masked-Attention Decoder: Each query carries two temporal states that share a content embedding. These states attend to their respective dates within the shared mask, then compare their readings via joint self-attention to recognize transitions.
The overall framework of MaskSCD, showing the Siamese encoder, fusion pyramid, and the specialized transformer decoder.

Datasets

The model was evaluated on three diverse benchmarks representing different sensors and resolutions:

  • SECOND: Aerial imagery (0.5 to 3 m resolution) with 7 land-cover classes.
  • Landsat-SCD: Medium-resolution (30 m) satellite imagery with 5 classes.
  • Hi-UCD: Ultra-high-resolution (0.1 m) aerial imagery with 10 classes, providing full-coverage annotations.

Key Insights and Ideas

  • Linear Label Space: A naive approach to SCD might try to classify every possible "from-to" transition. If you have 10 classes, that's 100 possible transitions ($K^2$). MaskSCD keeps the label space linear ($2 \times K$) by predicting classes for each date separately but within a shared spatial context.
  • Consistency by Design: By forcing the change map to be a derivative of the semantic class pair, the model inherently respects the logic of the scene.
  • Object Integrity: Because the model predicts masks (segments) rather than individual pixels, the resulting change maps have much sharper boundaries and better represent real-world objects like factory halls or agricultural parcels.
Figure 3: Bitemporal segments of a SECOND sample, the regions of constant class pair in the label-pair image. Changed segments are labeled in red.

Key Results

MaskSCD was compared against 14 recent SCD models (including CNN, state-space, and foundation-model-based approaches) and came out on top:

  • Superior Accuracy: It achieved the best separated kappa, semantic change $F_1$, and transition $F_1$ on all three datasets. It exceeded the strongest competitor by about 5 points in mean separated kappa.
  • Zero Contradictions: While competitors produced significant contradictory change pixels (especially on the complex Hi-UCD dataset), MaskSCD produced 0%.
  • Efficiency: Despite its high performance, MaskSCD is faster at inference than many state-space and query-based baselines.

Limitations and Future Work

  • Backbone Dependency: The model's performance scales with the size of the backbone (e.g., Swin-L vs. Swin-B). While accurate, using massive foundation models as encoders increases computational cost.
  • Time Series Extension: The current work focuses on bitemporal (two-date) change. The authors suggest the shared-mask formulation could be extended to image time series, where a segment would carry a "land-cover trajectory" over multiple acquisition dates.
  • Small Objects: Like many transformer-based architectures, very small or thin change regions might still pose a challenge for mask-based proposals compared to dense per-pixel methods.

The code for MaskSCD is available at https://github.com/EricYu97/MaskSCD.

Summary updated

Abstract

Semantic change detection (SCD) identifies where the land surface has changed and which land-cover transition has taken place. Most deep SCD networks predict the change map and the land-cover map of each date with separate per-pixel decoders. These outputs need not agree, so a pixel can be labeled as changed although it has the same class at both dates, and per-pixel decisions fragment objects. We present MaskSCD, which formulates SCD as the mask classification of a bitemporal label-pair image. Each bitemporal segment, a region of constant land-cover class pair, is predicted as one mask shared by both dates together with one class per date, and the change map is derived from the predicted class pair. This formulation keeps the change and semantic maps consistent and the label space linear in the number of classes. A bitemporal masked-attention transformer decoder implements it by reading each segment in both images and relating the two readings, and a bitemporal fusion pyramid provides per-date and change-aware features. MaskSCD is evaluated on SECOND, Landsat-SCD and Hi-UCD, which span 0.1 to 30 m resolution and four to nine land-cover classes, against fourteen recent SCD models re-trained under one protocol. MaskSCD achieves the best separated kappa, semantic change $F_1$ and transition $F_1$ on all three datasets and exceeds the strongest competitor by about five points in mean separated kappa. It produces no contradictory change pixels, which account for nearly 30% of the predicted change of some multi-task networks. The code is available at https://github.com/EricYu97/MaskSCD.
BibTeX
@misc{yu2026maskscd,
  author = {Weikang Yu and Pedram Ghamisi},
  title = {{MaskSCD: Bitemporal Mask Classification for Semantic Change Detection in Remote Sensing Images}},
  year = {2026},
  eprint = {2611.18350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18350},
}
Figure