TL;DR
The researchers introduce VirtualHAG (VirtualHome Action Genome), a large-scale simulated dataset for spatio-temporal scene graph generation. Unlike real-world datasets, VirtualHAG provides perfectly consistent, per-frame annotations. Alongside the dataset, they propose SGTracker, a tracking-based transformer model that utilizes temporal context to distinguish between visually similar but temporally distinct actions, such as "opening" versus "closing" a cabinet.
What Problem Does This Paper Address?
Spatio-temporal scene graphs are designed to map humans, objects, and their evolving relationships (e.g., "person holding cup," "person walking toward chair") across video frames. However, the field is currently held back by three major flaws in existing real-world datasets like Action Genome:
- Inconsistent Labeling: Human annotators often provide conflicting labels for nearly identical frames. For example, a person might be labeled as "in front of" a chair in one frame and "on the side of" it in the next, even if neither has moved.
- Sparse Annotations: Manual labeling is prohibitively expensive, leading many datasets to annotate only a few frames per second, which destroys the "dense" temporal flow necessary for learning complex actions.
- Lack of Temporal Reasoning: Most current benchmarks focus on relationships inferable from a single static image. They struggle with actions that require observing change over time, such as distinguishing "sitting down" from "standing up."
Methods
The researchers address these issues through a dual contribution: a high-quality synthetic dataset and a specialized model architecture.
1. VirtualHAG Dataset Generation
Using the VirtualHome simulator, the authors generated 2,588 indoor activity videos. Because the data is simulated, the ground truth—including object positions, states, and human actions—is known perfectly by the engine. This allowed for the automatic generation of dense, per-frame, and 100% consistent labels. Crucially, the dataset includes relationship pairs that explicitly require temporal context, such as grabbing/putting back and approaching/not approaching.
2. SGTracker (Scene Graph Tracker)
Standard models often treat video frames as a series of independent images. SGTracker is a transformer encoder-decoder framework designed to "track" relationships over time:
- Encoder: This component processes previous frames to establish the spatio-temporal context of moving objects.
- Decoder: A cross-attention mechanism associates the current frame's spatial information with the history stored in the encoder.
- Gaussian Masks: To improve localization, the model uses Gaussian-shaped masks derived from ground truth boxes. This focuses the transformer's attention specifically on the human and the object of interest.
Datasets
- VirtualHAG (New): Comprises 2,588 videos (~574k frames) featuring 50 object categories and 19 relationship types, captured from 4–8 camera viewpoints per scene.
- Action Genome: A standard real-world benchmark used to test "Sim2Real" transferability.
- HOMAGE: A real-world dataset used for qualitative comparisons of labeling errors.
Key Insights and Results
Simulation as a "Silver Bullet" for Labels
The paper demonstrates that simulation is not just a tool for generating more data, but for generating cleaner data. By eliminating human annotator bias and error, models can learn more precise boundaries for complex relationships.
Temporal Context is Mandatory
The authors show that without looking at previous frames, models are essentially guessing when trying to distinguish between actions like "opening" and "closing" a drawer. Temporal context provided by SGTracker significantly stabilizes these predictions.
Performance and Sim2Real Transfer
- State-of-the-Art Performance: On the VirtualHAG dataset, SGTracker significantly outperformed existing methods. In the Scene Graph Detection (SGDET) task—the most challenging task where the model must localize objects and predict relationships—SGTracker achieved a Recall@10 of 41.8, compared to 28.0 for the previous best method (RelDN).
- Sim2Real Boost: Pre-training on VirtualHAG boosted the Predicate Classification (PredCLS) Recall@20 on the real-world Action Genome dataset from 70.9% to 72.1%, proving that simulated relationships translate effectively to real-world scenarios.
Limitations and Future Work
While promising, the model currently requires known object regions in previous frames to start tracking effectively. Future work will focus on integrating a more robust "cold-start" object detector. Additionally, because the model processes object pairs individually, computational costs scale significantly with scene complexity. The authors suggest that a multi-object tracking framework could mitigate this scaling issue in the future.
Summary updated