Graph Representation for Order-aware Visual Transformation

Yue Qiu, Yanjun Sun, Fumiya Matsuzawa, Kenji Iwata, Hirokatsu KataokaCVPR 2023 · DOI

Abstract

Identifying temporal sequences in visual changes is a fundamental human capability often overlooked in computer vision. This paper introduces the Order-aware Visual Transformation (OVT) task, which requires models to determine both the nature and the chronological order of changes between image pairs. We propose VTGen, a framework that represents scene dynamics as a directed graph, effectively disentangling content from temporal flow. Experimental results on the new OVT dataset and existing benchmarks demonstrate that VTGen significantly outperforms traditional text-based change captioning models, setting a new state-of-the-art in complex visual reasoning.

Introduction

While existing AI models can detect differences between "before" and "after" images, they typically treat these changes as simultaneous. However, real-world scenarios often involve sequential transformations where the order of events is critical for understanding causality and spatial relationships. For instance, moving a chair before placing a rug requires different reasoning than the reverse.

Current methods often rely on natural language descriptions, which become increasingly convoluted as the number of changes grows. To address this, we define the Order-aware Visual Transformation (OVT) task and introduce VTGen, a model designed to output structured Visual Transformation Graphs.

The VTGen Framework

VTGen moves beyond simple list-based or sentence-based outputs by modeling scene changes as a directed graph. In this representation, nodes encapsulate the attributes of individual changes (e.g., "add," "delete," or "move"), while directed edges explicitly define the temporal sequence between them.

The VTGen architecture processes image pairs to generate a structured visual transformation graph where nodes represent changes and edges represent temporal order.

The architecture consists of three primary components:

  1. Transformation Encoder: A hybrid CNN-Transformer module that extracts features and identifies cross-image correlations.
  2. Transformation Graph Generator: This includes a Change Content Decoder to identify node attributes and a Change Order Decoder to predict the directed edges.
  3. Graph Matching Loss: A bipartite matching loss is employed to train the model effectively without requiring a fixed starting point for the graph generation.

The OVT Dataset and Benchmarking

To evaluate this new task, we introduce the Order-aware Visual Transformation (OVT) dataset. Generated using the CLEVR engine, it contains 150,000 image pairs featuring 15 distinct temporal patterns. These patterns range from simple synchronous changes to complex sequences of multiple asynchronous moves, providing a rigorous test for temporal reasoning.

Examples from the OVT dataset illustrating various temporal patterns, including synchronous and asynchronous changes, and their corresponding graph representations.

Key Results and Insights

VTGen demonstrates superior performance across multiple benchmarks:

  • Superior Accuracy: On the OVT dataset, VTGen achieved a per-scene accuracy of 57.8%, outperforming the previous best text-based models by 15.7%.
  • Structural Robustness: Unlike language-based models whose performance degrades sharply with increased scene complexity, VTGen maintains a higher accuracy floor, particularly when evaluating the correct order of changes.
  • Generalization: On the standard CLEVR-Multi-Change benchmark, VTGen established new state-of-the-art scores across metrics such as BLEU, CIDEr, and SPICE, proving its utility even for non-order-specific tasks.

A key insight from our study is that representing changes as graphs disentangles "what" happened from "when" it happened. This structural clarity allows for more transparent reasoning and better handling of complex spatial interactions, such as occlusions that occur during "add" or "move" operations.

Conclusion and Future Work

VTGen represents a significant step toward human-level visual reasoning by incorporating temporal order into change recognition. However, a "human-AI gap" of approximately 20% remains on the OVT dataset, indicating that current models still struggle with the most complex spatial-temporal sequences. Future research should focus on bridging this gap and extending these graph-based representations to real-world, non-synthetic environments like robotic assembly and automated scene monitoring.

Summary updated

BibTeX
@inproceedings{qiu2023graph,
  author = {Yue Qiu and Yanjun Sun and Fumiya Matsuzawa and Kenji Iwata and Hirokatsu Kataoka},
  title = {{Graph Representation for Order-aware Visual Transformation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02183},
}
Figure