VirtualHome Action Genome: A Simulated Spatio-Temporal Scene Graph Dataset with Consistent Relationship Labels

Yue Qiu, Yoshiki Nagasaki, Kensho Hara, Hirokatsu Kataoka, Ryota Suzuki, Kenji Iwata, Yutaka Satoh

TL;DR

The researchers introduce VirtualHAG (VirtualHome Action Genome), a large-scale simulated dataset for spatio-temporal scene graph generation. Unlike real-world datasets, VirtualHAG provides perfectly consistent, per-frame annotations. Alongside the dataset, they propose SGTracker, a tracking-based transformer model that utilizes temporal context to distinguish between visually similar but temporally distinct actions, such as "opening" versus "closing" a cabinet.

What Problem Does This Paper Address?

Spatio-temporal scene graphs are designed to map humans, objects, and their evolving relationships (e.g., "person holding cup," "person walking toward chair") across video frames. However, the field is currently held back by three major flaws in existing real-world datasets like Action Genome:

  1. Inconsistent Labeling: Human annotators often provide conflicting labels for nearly identical frames. For example, a person might be labeled as "in front of" a chair in one frame and "on the side of" it in the next, even if neither has moved.
  2. Sparse Annotations: Manual labeling is prohibitively expensive, leading many datasets to annotate only a few frames per second, which destroys the "dense" temporal flow necessary for learning complex actions.
  3. Lack of Temporal Reasoning: Most current benchmarks focus on relationships inferable from a single static image. They struggle with actions that require observing change over time, such as distinguishing "sitting down" from "standing up."
Figure 3: Comparison of relationship labels across frames, illustrating how subtle movements change the spatio-temporal scene graph.

Methods

The researchers address these issues through a dual contribution: a high-quality synthetic dataset and a specialized model architecture.

1. VirtualHAG Dataset Generation

Using the VirtualHome simulator, the authors generated 2,588 indoor activity videos. Because the data is simulated, the ground truth—including object positions, states, and human actions—is known perfectly by the engine. This allowed for the automatic generation of dense, per-frame, and 100% consistent labels. Crucially, the dataset includes relationship pairs that explicitly require temporal context, such as grabbing/putting back and approaching/not approaching.

2. SGTracker (Scene Graph Tracker)

Standard models often treat video frames as a series of independent images. SGTracker is a transformer encoder-decoder framework designed to "track" relationships over time:

  • Encoder: This component processes previous frames to establish the spatio-temporal context of moving objects.
  • Decoder: A cross-attention mechanism associates the current frame's spatial information with the history stored in the encoder.
  • Gaussian Masks: To improve localization, the model uses Gaussian-shaped masks derived from ground truth boxes. This focuses the transformer's attention specifically on the human and the object of interest.

Datasets

  • VirtualHAG (New): Comprises 2,588 videos (~574k frames) featuring 50 object categories and 19 relationship types, captured from 4–8 camera viewpoints per scene.
  • Action Genome: A standard real-world benchmark used to test "Sim2Real" transferability.
  • HOMAGE: A real-world dataset used for qualitative comparisons of labeling errors.
Figure 6: A complete action sequence in VirtualHAG (Grab, Walk, Open, Putback) showing dense and consistent relationship annotations across time.

Key Insights and Results

Simulation as a "Silver Bullet" for Labels

The paper demonstrates that simulation is not just a tool for generating more data, but for generating cleaner data. By eliminating human annotator bias and error, models can learn more precise boundaries for complex relationships.

Temporal Context is Mandatory

The authors show that without looking at previous frames, models are essentially guessing when trying to distinguish between actions like "opening" and "closing" a drawer. Temporal context provided by SGTracker significantly stabilizes these predictions.

Performance and Sim2Real Transfer

  • State-of-the-Art Performance: On the VirtualHAG dataset, SGTracker significantly outperformed existing methods. In the Scene Graph Detection (SGDET) task—the most challenging task where the model must localize objects and predict relationships—SGTracker achieved a Recall@10 of 41.8, compared to 28.0 for the previous best method (RelDN).
  • Sim2Real Boost: Pre-training on VirtualHAG boosted the Predicate Classification (PredCLS) Recall@20 on the real-world Action Genome dataset from 70.9% to 72.1%, proving that simulated relationships translate effectively to real-world scenarios.

Limitations and Future Work

While promising, the model currently requires known object regions in previous frames to start tracking effectively. Future work will focus on integrating a more robust "cold-start" object detector. Additionally, because the model processes object pairs individually, computational costs scale significantly with scene complexity. The authors suggest that a multi-object tracking framework could mitigate this scaling issue in the future.

Summary updated

Abstract

Spatio-temporal scene graph generation is an essential task in household activity recognition that aims to identify human-object interactions. Constructing a dataset with per-frame object region and consistent relationship annotations requires extremely high labor costs. Existing datasets sparsely annotate frames sampled from videos, resulting in the lack of dense spatio-temporal correlation in videos. Additionally, existing datasets contain inconsistent relationship annotations, leading to the problem of learning ambiguous temporal associations. Moreover, existing datasets mainly discuss relationships that can be inferred from a single frame, ignoring the significance of temporal associations. To resolve those issues, we created a simulated dataset with per-frame consistent annotations and introduced a range of relationships requiring both spatial and temporal context. Most existing methods explore spatial correlations within single images and do not explicitly consider the dynamic changes across frames. Therefore, we proposed a tracking-based approach that explicitly grasps spatio-temporal human-object interactions while simultaneously localizing humans and objects. Our proposed approach achieved state-of-the-art performance on scene graph generation and outperformed existing methods in scene graph localization by large margins on the proposed dataset. Moreover, the experiments show the efficacy of pre-training on the proposed dataset while adapting to a previous benchmark consisting of real daily videos, indicating the potential of the proposed dataset in real-world scenarios.
BibTeX
@inproceedings{qiu2023virtualhome,
  author = {Yue Qiu and Yoshiki Nagasaki and Kensho Hara and Hirokatsu Kataoka and Ryota Suzuki and Kenji Iwata and Yutaka Satoh},
  title = {{VirtualHome Action Genome: A Simulated Spatio-Temporal Scene Graph Dataset with Consistent Relationship Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00335},
}
Figure