Abstract
Identifying temporal sequences in visual changes is a fundamental human capability often overlooked in computer vision. This paper introduces the Order-aware Visual Transformation (OVT) task, which requires models to determine both the nature and the chronological order of changes between image pairs. We propose VTGen, a framework that represents scene dynamics as a directed graph, effectively disentangling content from temporal flow. Experimental results on the new OVT dataset and existing benchmarks demonstrate that VTGen significantly outperforms traditional text-based change captioning models, setting a new state-of-the-art in complex visual reasoning.
Introduction
While existing AI models can detect differences between "before" and "after" images, they typically treat these changes as simultaneous. However, real-world scenarios often involve sequential transformations where the order of events is critical for understanding causality and spatial relationships. For instance, moving a chair before placing a rug requires different reasoning than the reverse.
Current methods often rely on natural language descriptions, which become increasingly convoluted as the number of changes grows. To address this, we define the Order-aware Visual Transformation (OVT) task and introduce VTGen, a model designed to output structured Visual Transformation Graphs.
The VTGen Framework
VTGen moves beyond simple list-based or sentence-based outputs by modeling scene changes as a directed graph. In this representation, nodes encapsulate the attributes of individual changes (e.g., "add," "delete," or "move"), while directed edges explicitly define the temporal sequence between them.
The architecture consists of three primary components:
- Transformation Encoder: A hybrid CNN-Transformer module that extracts features and identifies cross-image correlations.
- Transformation Graph Generator: This includes a Change Content Decoder to identify node attributes and a Change Order Decoder to predict the directed edges.
- Graph Matching Loss: A bipartite matching loss is employed to train the model effectively without requiring a fixed starting point for the graph generation.
The OVT Dataset and Benchmarking
To evaluate this new task, we introduce the Order-aware Visual Transformation (OVT) dataset. Generated using the CLEVR engine, it contains 150,000 image pairs featuring 15 distinct temporal patterns. These patterns range from simple synchronous changes to complex sequences of multiple asynchronous moves, providing a rigorous test for temporal reasoning.
Key Results and Insights
VTGen demonstrates superior performance across multiple benchmarks:
- Superior Accuracy: On the OVT dataset, VTGen achieved a per-scene accuracy of 57.8%, outperforming the previous best text-based models by 15.7%.
- Structural Robustness: Unlike language-based models whose performance degrades sharply with increased scene complexity, VTGen maintains a higher accuracy floor, particularly when evaluating the correct order of changes.
- Generalization: On the standard CLEVR-Multi-Change benchmark, VTGen established new state-of-the-art scores across metrics such as BLEU, CIDEr, and SPICE, proving its utility even for non-order-specific tasks.
A key insight from our study is that representing changes as graphs disentangles "what" happened from "when" it happened. This structural clarity allows for more transparent reasoning and better handling of complex spatial interactions, such as occlusions that occur during "add" or "move" operations.
Conclusion and Future Work
VTGen represents a significant step toward human-level visual reasoning by incorporating temporal order into change recognition. However, a "human-AI gap" of approximately 20% remains on the OVT dataset, indicating that current models still struggle with the most complex spatial-temporal sequences. Future research should focus on bridging this gap and extending these graph-based representations to real-world, non-synthetic environments like robotic assembly and automated scene monitoring.
Summary updated