Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning

Jin Chen, Xiaofeng Ji, Xinxiao WuAAAI 2022 · OpenReview

Abstract

Scene graph in a video conveys a wealth of information about objects and their relationships in the scene, thus benefiting many downstream tasks such as video captioning and visual question answering. Existing methods of scene graph generation require large-scale training videos annotated with objects and relationships in each frame to learn a powerful model. However, such comprehensive annotation is time-consuming and labor-intensive. On the other hand, it is much easier and less cost to annotate images with scene graphs, so we investigate leveraging annotated images to facilitate training a scene graph generation model for unannotated videos, namely image-to-video scene graph generation. This task presents two challenges: 1) infer unseen dynamic relationships in videos from static relationships in images due to the absence of motion information in images; 2) adapt objects and static relationships from images to video frames due to the domain shift between them. To address the first challenge, we exploit external commonsense knowledge to infer the unseen dynamic relationship from the temporal evolution of static relationships. We tackle the second challenge by hierarchical adversarial learning to reduce the data distribution discrepancy between images and video frames. Extensive experiment results on two benchmark video datasets demonstrate the effectiveness of our method.

Introduction

A Scene Graph is a structured representation of a visual scene where objects are nodes and their interactions are edges (e.g., {person, ride, bicycle}). While these graphs are vital for high-level reasoning in AI, generating them for video is exceptionally difficult. Traditional supervised methods require frame-by-frame annotations, which are prohibitively expensive to produce at scale.

This paper proposes an "image-to-video" transfer approach, leveraging abundant annotated image datasets (like Visual Genome) to train models for unannotated videos. This transfer faces two primary hurdles: the Temporal Gap, where static images lack motion-based relationships like "moving past," and the Domain Shift, where video frames often suffer from lower quality and motion blur compared to curated static images.

Methodology

The proposed framework addresses these challenges through a multi-stage architecture involving adversarial adaptation and linguistic knowledge reasoning.

Overview of the proposed framework including cross-domain object detection, static relationship prediction, and dynamic relationship learning.

Hierarchical Adversarial Learning

To bridge the domain gap, the model employs two levels of adversarial training using Gradient Reversal Layers (GRL):

  1. Image-level Alignment: Aligns global features such as style and illumination between source images and target video frames.
  2. Instance-level Alignment: Ensures that specific object proposals (e.g., a "person") yield domain-invariant features, regardless of whether they appear in a crisp image or a blurry video frame.

Knowledge Reasoning for Dynamic Relationships

To overcome the lack of motion in static training data, the authors treat dynamic actions as the temporal evolution of static states. They developed a Relationship Commonsense Base (RCB) using external text sources like Wikipedia. This base defines dynamic predicates as sequences of static ones.

Conceptual illustration of inferring a dynamic relationship ('ride past') from a sequence of static relationships.

As shown in the figure above, the model observes a sequence of static relationships—ride front, ride left, and ride behind—and uses an LSTM to map these visual embeddings to the language embedding of the dynamic predicate ride past in a shared visual-language space.

Experimental Results

The method was evaluated using an unsupervised domain adaptation protocol, training on Visual Genome (images) and testing on VidVRD and VidOR (videos).

  • Dynamic Relationship Detection: On the VidVRD dataset, the model achieved a 13.64% Precision@1 for dynamic relationships. Crucially, baseline models trained on images without the knowledge reasoning module failed completely (0% precision) on these motion-dependent predicates.
  • Annotation Efficiency: The approach achieved competitive performance using only 1,572 annotated images, compared to supervised methods that typically require tens of thousands of annotated video frames.
  • Ablation Studies: Results confirmed that both hierarchical adversarial learning and the commonsense reasoning module are essential; removing either component resulted in a significant performance drop.

Conclusion

By combining adversarial domain adaptation with linguistic commonsense, this framework successfully transfers knowledge from static images to dynamic video environments. This significantly reduces the annotation burden for Scene Graph Generation. Future work may focus on automated discovery of commonsense rules from unlabeled video data and expanding the model to handle more complex, multi-object interactions over longer durations.

Summary updated

Abstract

Scene graph in a video conveys a wealth of information about objects and their relationships in the scene, thus benefiting many downstream tasks such as video captioning and visual question answering. Existing methods of scene graph generation require large-scale training videos annotated with objects and relationships in each frame to learn a powerful model. However, such comprehensive annotation is time-consuming and labor-intensive. On the other hand, it is much easier and less cost to annotate images with scene graphs, so we investigate leveraging annotated images to facilitate training a scene graph generation model for unannotated videos, namely image-to-video scene graph generation. This task presents two challenges: 1) infer unseen dynamic relationships in videos from static relationships in images due to the absence of motion information in images; 2) adapt objects and static relationships from images to video frames due to the domain shift between them. To address the first challenge, we exploit external commonsense knowledge to infer the unseen dynamic relationship from the temporal evolution of static relationships. We tackle the second challenge by hierarchical adversarial learning to reduce the data distribution discrepancy between images and video frames. Extensive experiment results on two benchmark video datasets demonstrate the effectiveness of our method.
BibTeX
@inproceedings{chen2022adaptive,
  author = {Jin Chen and Xiaofeng Ji and Xinxiao Wu},
  title = {{Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=l70-Dys8I5t},
}
Figure