Introduction
Image change captioning (ICC) is the task of describing the differences between two similar images in natural language. Unlike standard image captioning, which describes a single static scene, ICC requires a model to ignore "distractor" similarities and focus on subtle variations—such as a bird's beak changing from orange to yellow or a tail becoming slightly longer.
The authors identify two major gaps in existing methods:
- Coarse Perception: Most models treat image features as a whole, failing to capture the "fine-grained" details (color, shape, texture) necessary for precise descriptions.
- Localization Failure: Models often struggle to "find" exactly where the change occurred, leading to captions that are either vague or hallucinate changes in the wrong areas.
Methodology
The proposed Find and Perceive (F&P) paradigm consists of two primary modules integrated into a Transformer-based architecture:
Fine-Grained Semantic Change Perception (FGSCP)
This module improves the "Perceive" aspect through two stages:
- Fine-Grained Feature Learning (FGFL): To force the model to learn subtle patterns, an auxiliary classification task is introduced. For example, in a bird dataset, the model must first learn to categorize the bird species. This ensures the underlying features are rich enough to distinguish between minute anatomical details.
- Difference Enhancement (DE): This uses a cross-attention mechanism between the two images. Because unchanged regions will have high similarity and changed regions will have low similarity, the cross-attention effectively acts as a "difference engine," highlighting the areas that have diverged.
Weakly-Supervised Discriminator (WSD)
This addresses the "Find" aspect without requiring expensive, manually-labeled bounding boxes. The authors use an "incomplete" version of the model to generate an initial caption. They then analyze the attention weights—identifying which parts of the image the model focused on to produce specific words. These weights are converted into "pseudo-labels" (0 for unchanged, 1 for changed) to train a discriminator that teaches the final model exactly where to focus its attention.
Datasets and Evaluation
The model was evaluated on four diverse datasets:
- Birds-to-Words: 41k sentences describing fine-grained changes in bird photos.
- CLEVR-Change: A large-scale synthetic dataset (79.6k image pairs) with changes in color, shape, and position.
- Spot-the-Diff: 13k image pairs from surveillance cameras focusing on content changes.
- Image-Editing-Request: 3.9k image pairs with instructions produced by manual image editing.
Key Results
The F&P model achieved state-of-the-art (SOTA) results across all major metrics:
- Birds-to-Words: Achieved a 33.4 BLEU-4 and 28.1 CIDEr, significantly outperforming the previous SOTA (PLC) which scored 31.0 and 25.3, respectively.
- CLEVR-Change: Reached a 56.4 BLEU-4 and 129.0 CIDEr. When combined with CLIP features, these numbers improved to 58.9 and 153.6.
- Qualitative Success: Visualization of the attention maps showed that the F&P model focuses much more tightly on the actual changed regions compared to baseline models, which often had "blurry" attention spread across the entire image.
Conclusion
The "Find and Perceive" framework advances image change captioning by explicitly separating localization from feature extraction. By treating classification as a feature booster and using internal attention as a supervision signal, the model avoids the "black box" trap and provides more precise descriptions. Future work will explore scaling this framework to handle more complex, multi-object scene changes and integrating advanced vision-language pre-training (VLP) models.
Summary updated