RealGraph: A Multiview Dataset for 4D Real-world Context Graph Generation

Haozhe Lin, Zequn Chen, Jinzhi Zhang, Bing Bai, Yu Wang, Ruqi Huang, Lu Fang

RealGraph: Mapping the 4D World Through Multiview Vision

Researchers from Tsinghua University have introduced RealGraph, a massive multiview dataset and a new task called Context Graph Generation (CGG). By moving beyond static 3D scene graphs, this work enables AI to track objects and their semantic relationships (like "person holding cake") across both 3D space and time (4D) using only standard synchronized 2D cameras.

The Challenge of Dynamic 3D Understanding

While AI has become adept at 2D object detection, understanding a complex, dynamic 3D environment remains a significant challenge for robotics and embodied AI. Existing 3D scene graphs are largely static—mapping a room's layout—or rely on expensive active sensors like LiDAR or RGB-D cameras, which are difficult to deploy in every real-world scenario.

Human activities are inherently 4D: they occur in 3D space over a duration of time. To truly understand a scene, such as a waiter navigating a busy café, an AI needs to know not just where a cup is, but who is holding it, how that relationship changes as they move, and it must maintain this understanding even when objects are occluded in one camera view but visible in another.

Overview of the RealGraph task: generating a 4D context graph from synchronized multiview 2D video.

Methodology: MCGNet

The authors propose a new paradigm called Context Graph Generation (CGG). Unlike traditional tracking, CGG aims to produce a spatio-temporal graph where nodes are objects (with 3D coordinates and consistent IDs) and edges are semantic relationships. To solve this, they developed MCGNet (Multiview Context Graph inference Network), a multi-stage baseline architecture.

The MCGNet architecture: projecting 2D features into a 3D volume for joint detection, relationship inference, and tracking.

MCGNet operates through several key stages:

  1. Feature Extraction: Standard 2D backbones extract features from every camera view.
  2. Back-Projection: These 2D features are projected into a shared 3D voxel volume based on camera calibrations.
  3. Feature Fusion (FF): A novel multi-scale 3D CNN refines this volume, helping the model detect small objects (like phones or cups) that might be lost in a single-scale 3D representation.
  4. Detection & Relation Heads: The model predicts 3D bounding boxes and uses a Bi-LSTM to infer relationships (e.g., ⟨man — sit at — table⟩) by analyzing the features of object pairs over time.
  5. Tracking: A Kalman filter combined with a "double association" scheme ensures that object IDs remain consistent even during temporary occlusions.

The RealGraph Dataset

The core contribution is the RealGraph dataset, the first of its kind tailored for 4D context graphs. It features 13 real-world scenes (cafés, offices) captured by 8–15 synchronized GoPro cameras, totaling over 2.4 million video frames. The dataset includes 2.3M 2D boxes, 420K 3D boxes, and 130K 3D relationship labels across 37 object categories and 18 relationship types. Unlike autonomous driving datasets, RealGraph focuses on complex human-object interactions in indoor and outdoor social settings.

Key Insights and Results

  • Consistency is Key: A "Context Graph" must be consistent across both space (all camera views agree) and time (IDs remain stable). The authors introduced Context Graph Recall (CGR) to specifically penalize identity switches.
  • The Power of Multiview: By using 8+ cameras, the system compensates for "out-of-view" problems. If a person is hidden in one view, the 3D feature volume still captures them via other angles.
  • Small Object Recovery: The Feature Fusion (FF) module improved mean Average Precision (mAP) by 3.3% overall, with significant gains for difficult, small-scale categories like "cup."
  • Tracking Performance: The "double association" strategy improved Multiple Object Tracking Accuracy (MOTA) by 0.11, proving that smart data association is as critical as detection power.

Limitations and Future Work

MCGNet is currently a multi-stage pipeline, which is prone to error accumulation—if the detector misses a box, the relationship head cannot recover. Future work will likely move toward end-to-end joint optimization, where detection, tracking, and relationship reasoning happen simultaneously. Additionally, the computational cost of processing high-resolution multiview 4D volumes suggests a need for more efficient sparse-convolution architectures.

Summary updated

Abstract

Understanding 4D scene context in real world has become urgently critical for deploying sophisticated AI systems. In this paper, we propose a brand new scene understanding paradigm called "Context Graph Generation (CGG)", aiming at abstracting holistic semantic information in the complicated 4D world. The CGG task capitalizes on the calibrated multiview videos of a dynamic scene, and targets at recovering semantic information (coordination, trajectories and relationships) of the presented objects in the form of spatio-temporal context graph in 4D space. We also present a benchmark 4D video dataset "RealGraph", the first dataset tailored for the proposed CGG task. The raw data of RealGraph is composed of calibrated and synchronized multiview videos. We exclusively provide manual annotations including object 2D&3D bounding boxes, category labels and semantic relationships. We also make sure the annotated ID for every single object is temporally and spatially consistent. We propose the first CGG baseline algorithm, Multiview-based Context Graph Generation Network (MCGNet), to empirically investigate the legitimacy of CGG task on RealGraph dataset. We nevertheless reveal the great challenges behind this task and encourage the community to explore beyond our solution. Our project page is at https://github.com/THU-luvision/RealGraph.
BibTeX
@inproceedings{lin2023realgraph,
  author = {Haozhe Lin and Zequn Chen and Jinzhi Zhang and Bing Bai and Yu Wang and Ruqi Huang and Lu Fang},
  title = {{RealGraph: A Multiview Dataset for 4D Real-world Context Graph Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00348},
}
Figure