Describing and Localizing Multiple Changes with Transformers

Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, Yutaka Satoh

TL;DR

This paper tackles the challenge of describing and localizing multiple changes in image pairs, a more realistic scenario than single-change detection. They introduce a novel transformer-based framework, MCCFormers, that effectively identifies and describes these multiple changes, outperforming existing methods on both new and existing benchmarks.

What Problem Does This Paper Address?

Change captioning, the task of describing what has changed between two images, is crucial for applications like urban planning, disaster management, and robotics. While existing methods have focused on describing a single change, real-world scenarios often involve multiple simultaneous changes within a scene. Accurately detecting, localizing, and describing these multiple changes, especially when the number of changes is unknown beforehand, remains a significant challenge. Previous approaches often struggle to disentangle and describe these concurrent alterations, leading to incomplete or inaccurate captions.

Methods

The core of this work is the MCCFormers framework, a transformer-based encoder-decoder model.

  • Feature Extraction: Standard CNNs are used to extract features from both the "before" and "after" images.
  • Change Encoder: This is where the novelty lies. MCCFormers employs transformer encoders to capture dense correlations between image regions across the two input images. Two variants are proposed:
  • MCCFormers-D (Dual Encoder): Uses two transformer encoders with shared weights to capture inter-image relevance.
  • MCCFormers-S (Single Encoder): Uses a single transformer encoder that concatenates features from both images, allowing it to capture both inter- and intra-image patch relationships.
  • Paragraph Decoder: A standard transformer decoder is used to generate a paragraph of descriptions. Crucially, this decoder dynamically attends to different image regions for each generated word, allowing it to distinguish and describe individual changes within the scene. This contrasts with prior methods that often generate a single attention map for the entire paragraph.

Datasets

  • CLEVR-Multi-Change Dataset: A novel dataset introduced by the authors to specifically address multi-change captioning and localization. It consists of image pairs with multiple changes, corresponding change captions, and bounding boxes for the changed regions. The dataset is generated from the CLEVR engine and includes scenes with one to four changes.
  • Spot-the-Diff Dataset: An existing dataset used for evaluating the proposed methods on scenarios with multiple changes.
  • CLEVR-Change Dataset: An existing dataset used for evaluating the methods on single-change captioning.

Key Insights and Ideas

  • Transformer Power for Change Detection: The paper demonstrates that transformer architectures, particularly their multi-head attention mechanism, are highly effective at capturing dense correlations between image regions, which is essential for identifying subtle or complex changes.
  • Dynamic Attention for Multi-Change: The key to handling multiple changes lies in the decoder's ability to dynamically attend to different image regions for each generated word. This allows the model to disentangle and describe each change individually, rather than producing a single, potentially confused, description.
  • Localization is Key: The ability to localize changes is as important as describing them. The proposed MCCFormers framework, especially MCCFormers-D, shows promising results in localizing changed regions, which is crucial for applications like robot manipulation.
  • Dataset Matters: The creation of the CLEVR-Multi-Change dataset is a significant contribution, providing a much-needed benchmark for evaluating multi-change captioning and localization capabilities.

Key Results

  • CLEVR-Multi-Change Performance: MCCFormers-S achieved the highest BLEU-4 score (+7.2 over previous methods) on the CLEVR-Multi-Change dataset. Both MCCFormers-D and MCCFormers-S significantly outperformed DUDA and M-VAM across various metrics (BLEU-4, CIDEr, METEOR, SPICE).
  • Localization Accuracy: MCCFormers-D achieved an overall localization accuracy of 53.9% on the CLEVR-Multi-Change dataset, significantly outperforming DUDA (40.0%).
  • Sentence Number Accuracy: MCCFormers-D achieved 92.5% accuracy in correctly generating the number of sentences for multi-change scenarios, with an average error (MAE) of less than 1.
  • Spot-the-Diff and CLEVR-Change: MCCFormers achieved comparable or superior results to state-of-the-art methods on these existing datasets, demonstrating its generalizability.

Limitations and Future Work

  • Viewpoint Change Robustness: The authors note that while their method shows promise, further investigation into robustness against larger viewpoint changes is warranted.
  • Interpretability of MCCFormers-S: The attention maps from MCCFormers-S tend to focus on both related and unrelated regions, potentially weakening interpretability compared to MCCFormers-D.
  • Dataset Size: The Spot-the-Diff dataset, while useful, might be too small to fully leverage the potential of transformer-based methods, suggesting a need for larger-scale datasets for future research.
  • Future Work: The paper suggests exploring change captioning on large-scale real-world imagery as a future research direction.

Summary updated

Abstract

Change captioning tasks aim to detect changes in image pairs observed before and after a scene change and generate a natural language description of the changes. Existing change captioning studies have mainly focused on a single change. However, detecting and describing multiple changed parts in image pairs is essential for enhancing adaptability to complex scenarios. We solve the above issues from three aspects: (i) We propose a simulation-based multi-change captioning dataset; (ii) We benchmark existing state-of-the-art methods of single change captioning on multi-change captioning; (iii) We further propose Multi-Change Captioning transformers (MCCFormers) that identify change regions by densely correlating different regions in image pairs and dynamically determines the related change regions with words in sentences. The proposed method obtained the highest scores on four conventional change captioning evaluation metrics for multi-change captioning. Additionally, our proposed method can separate attention maps for each change and performs well with respect to change localization. Moreover, the proposed framework outperformed the previous state-of-the-art methods on an existing change captioning benchmark, CLEVR-Change, by a large margin (+6.1 on BLEU-4 and +9.7 on CIDEr scores), indicating its general ability in change captioning tasks. The code and dataset are available at the project page <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{qiu2021describing,
  author = {Yue Qiu and Shintaro Yamamoto and Kodai Nakashima and Ryota Suzuki and Kenji Iwata and Hirokatsu Kataoka and Yutaka Satoh},
  title = {{Describing and Localizing Multiple Changes with Transformers}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00198},
}
Figure