Abstract: Change captioning involves describing semantic differences between two images of the same scene. A significant challenge in this task is viewpoint shift, where changes in camera perspective can dwarf actual semantic changes in pixel space. We propose Viewpoint-Agnostic Change Captioning (VACC), a framework that leverages a novel difference encoder and a cycle consistency module to robustly identify and describe changes without requiring external metadata like camera poses or depth maps. Our approach achieves state-of-the-art results on both synthetic and real-world datasets, demonstrating significant improvements in handling extreme viewpoint shifts.
Introduction
The task of change captioning requires a model to compare a "before" and "after" image and generate a natural language description of the semantic changes (e.g., "the blue cube was removed"). However, when these images are captured from different viewpoints, the resulting pixel-level differences are dominated by perspective shifts rather than actual scene changes. Traditional methods often rely on 3D point clouds or precise camera coordinates to align images, which are rarely available in real-world scenarios. VACC addresses this by learning to align features and verify captions using only RGB input.
Methodology
The VACC architecture is built around three core components designed to handle viewpoint invariance and ensure caption accuracy.
- Difference Encoder: This module bypasses simple pixel subtraction. It first uses Viewpoint Encoding to compute a similarity map between images, effectively "finding" corresponding objects across different perspectives. A Fused Difference module then isolates semantic changes (e.g., color shifts or object disappearance) while filtering out noise from the viewpoint change.
- Caption Generator: An attention-based LSTM processes the combined features from the "before" image, "after" image, and the calculated difference to generate the final description.
- Cycle Consistency Module: This module introduces a self-verification step. It treats the generated caption as a transformation operator. By attempting to reconstruct the "after" image features from the "before" image using the caption, the model is forced to produce descriptions that are grounded in the actual visual evidence.
Experiments and Results
The model was evaluated on CLEVR-Change, Spot-the-Diff, and a new dataset, CLEVR-DC, which features extreme viewpoint shifts where the camera moves to random positions in 3D space.
Key findings include:
- State-of-the-Art Performance: On the CLEVR-DC dataset, VACC improved the CIDEr score from 56.7 (previous best) to 71.7.
- Robustness to Perspective: As shown in the performance graphs, VACC maintains stable accuracy even as the viewpoint shift increases, whereas baseline models experience sharp declines.
- Generalizability: The Cycle Consistency (CC) module can be integrated into existing architectures (e.g., DUDA), providing a significant performance boost (e.g., +5.3 CIDEr points) as a standalone regularizer.
- Real-World Application: VACC achieved top scores on the Spot-the-Diff dataset (CIDEr 41.5), proving its effectiveness on natural, non-synthetic imagery.
Conclusion
VACC demonstrates that viewpoint-agnostic change captioning is achievable using only RGB images through intelligent feature alignment and cycle-consistent verification. While currently tested primarily on synthetic data and single-change scenarios, the framework provides a robust foundation for future work in automated surveillance, robotics, and visual assistance. Future research will explore scaling the similarity map calculations for high-resolution imagery and handling multiple simultaneous changes.
Summary updated