Anchoring with Visual Evidence: Explicit Semantic Anchors for Joint Change Detection and Captioning

Kai Zheng, Hang-Cheng Dong, Puyi Fang, Junjia Pan, Zhenkai Wu, Wei ZhangarXiv 2026 · arXiv

TL;DR

The Explicit Semantic Anchors (ESA) framework bridges the gap between pixel-level change detection and natural language captioning in remote sensing. By introducing a supervised interface that extracts interpretable factors—like entity type, temporal appearance, and spatial extent—the model creates a "semantic bridge" that significantly improves both the accuracy and the explainability of change descriptions.

What Problem Does This Paper Address?

In remote sensing, we often want to know two things about a pair of images taken at different times: where did things change (Change Detection, CD) and what actually happened (Change Captioning, CC).

The challenge lies in the "translation" process. Change detection produces dense, pixel-level maps, while captioning requires a compact set of high-level attributes (e.g., "new houses were built"). Most existing models treat this as a black-box mapping, making it hard to know if the model is actually "looking" at the right evidence when it writes a sentence. There is a need for an intermediate representation that is both grounded in visual evidence and easy for humans to inspect.

Methods

The authors propose Explicit Semantic Anchors (ESA), a supervised interface that sits between the visual encoder and the language decoder.

Figure 1. Overview of Explicit Semantic Anchors (ESA). A shared visual pathway predicts the change map and supplies dense visual memory. A prediction-guided, stop-gradient readout exposes change evidence to three factor heads for entity presence, temporal appearance variation, and spatial extent. The context route maps the three factor activations to three factor-specific tokens and appends them, separately and without aggregation, to the dense visual memory; the decoder can attend to each factor independently. The lexicon route maps the same activations to an additive bias on the output vocabulary logits before the softmax. Dashed arrows indicate auxiliary supervision derived from existing masks and image intensities during training; inference requires only the image pair.

Key components include:

  • Three Semantic Factors: The model explicitly predicts three attributes:
  • Entity Presence: Which categories (buildings, roads, etc.) changed?
  • Temporal Appearance: Did the area get brighter or darker? (A proxy for visual change).
  • Spatial Extent: Is the change small, medium, or large in terms of area?
  • Prediction-Guided Readout: Instead of just using raw image features, the model uses its own predicted change maps to "pool" relevant visual evidence into these factor heads.
  • Dual-Route Conditioning: These factors influence the captioning process through two paths:
  • Context Route: The factors are converted into "memory tokens" that the decoder can attend to, just like it attends to image pixels.
  • Lexicon Route: The factors provide a direct "vocabulary bias," making words related to the detected changes (like "building" or "large") more likely to be selected during sentence generation.
  • Zero-Cost Supervision: Crucially, the targets for these anchors are derived automatically from existing masks and image intensities. No extra human labeling is required.

Datasets

The model was evaluated on two major benchmarks for Multi-level Change Interpretation (MCI):

  • LEVIR-MCI: Contains over 10,000 bi-temporal image pairs with pixel-level masks and five reference captions per pair.
  • WHU-CDC: A similar dataset focused on building changes, used to test the model's robustness across different environments.

Key Insights and Ideas

  • Interpretability as a Feature: By forcing the model to predict explicit factors (Entity, Time, Space), researchers can inspect the "thoughts" of the model before it generates a caption. If the model says "houses" but the Entity Anchor predicted "roads," you know exactly where the reasoning failed.
  • The Power of Vocabulary Bias: While most models only give the decoder "more data" (context), ESA directly influences the "word choice" (lexicon). This ensures that the high-level semantic conclusions reached by the visual system are directly reflected in the final text.
  • Information Preservation: Even though the anchors provide high-level summaries, the model still retains access to "dense visual memory." This allows the captioner to describe fine-grained details (like the specific arrangement of houses) that the global anchors might miss.

Key Results

ESA achieved state-of-the-art performance on both benchmarks, showing that explicit anchoring helps the model "stay on track."

  • LEVIR-MCI: The model achieved a BLEU-4 score of 70.57, outperforming the previous best (CoDeRNet) by a significant 4.3 points. It also showed the highest CIDEr-D score (145.05), indicating its captions are much closer to human-written references.
  • WHU-CDC: It reached a 94.86 mIoU for detection and a 169.97 CIDEr-D for captioning, the highest values reported among compared methods.

Qualitatively, the model is much better at identifying the principal changed entities and their spatial arrangements, as seen in the visual comparisons.

Visual comparison of results. ESA (Ours) correctly identifies building and road changes and describes their spatial relationship more accurately than baseline models.

Limitations and Future Work

  • Global vs. Local: Currently, the anchors are "global" factors for the whole scene. They don't yet assign specific events to individual objects (e.g., "this specific building was demolished while that one was built").
  • Simple Temporal Cues: The temporal factor uses a basic brightness/intensity proxy. While effective, it doesn't capture complex semantic events like "vegetation growth" vs. "construction" directly.
  • Future Directions: The authors suggest extending this interface to object-level representations, which would allow for even more detailed and grounded claims in the generated captions.

Summary updated

Abstract

Joint change detection and captioning seeks to localize changes and describe them from bi-temporal remote sensing images. A key challenge is translating dense visual evidence into the semantic conditions that guide description generation. We propose Explicit Semantic Anchors (ESA), a supervised interface that exposes entity presence, temporal appearance variation, and spatial extent. A prediction-guided readout combines visual features and predicted change probabilities, and three factor heads estimate the corresponding anchor activations. These activations guide generation through two complementary routes. The context route maps the three factor activations to three separate memory tokens, one per factor, and appends them to the dense visual memory, while the lexicon route produces an additive vocabulary bias at each decoding step. Dense visual memory remains available for spatial detail and information beyond the explicit factors. Anchor targets are derived from existing images and change annotations, requiring no additional anchor-level language annotation. At inference, all conditions are computed from the input image pair and the model's own predictions. Experiments on LEVIR-MCI and WHU-CDC show strong joint detection and captioning performance, supporting the use of supervised visual attributes as explicit conditions for change description generation.
BibTeX
@misc{zheng2026anchoring,
  author = {Kai Zheng and Hang-Cheng Dong and Puyi Fang and Junjia Pan and Zhenkai Wu and Wei Zhang},
  title = {{Anchoring with Visual Evidence: Explicit Semantic Anchors for Joint Change Detection and Captioning}},
  year = {2026},
  eprint = {2611.24462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24462},
}
Figure