Holistic Pose Graph: Modeling Geometric Structure among Objects in a Scene using Graph Inference for 3D Object Prediction

Jiwei Xiao, Ruiping Wang, Xilin ChenICCV 2021 · OpenReview

Abstract

Due to the missing depth cues, it is essentially ambiguous to detect 3D objects from a single RGB image. Existing methods predict the 3D pose for each object independently or merely by combining local relationships within limited surroundings, but rarely explore the inherent geometric relationships from a global perspective. To address this issue, we argue that modeling geometric structure among objects in a scene is very crucial, and thus elaborately devise the Holistic Pose Graph (HPG) that explicitly integrates all geometric poses including the object pose treated as nodes and the relative pose treated as edges. The inference of the HPG uses GRU to encode the pose features from their corresponding regions in a single RGB image, and passes messages along the graph structure iteratively to improve the predicted poses. To further enhance the correspondence between the object pose and the relative pose, we propose a novel consistency loss to explicitly measure the deviations between them. Finally, we apply Holistic Pose Estimation (HPE) to jointly evaluate both the independent object pose and the relative pose. Our experiments on the SUN RGB-D dataset demonstrate that the proposed method provides a significant improvement on 3D object prediction.

Introduction

When humans view a 2D image of a room, they intuitively understand the spatial layout—that a chair is tucked under a desk or a lamp sits atop a nightstand. However, for computer vision models, monocular 3D object detection is a "broken" problem. Without depth sensors, there are infinite ways a 2D bounding box could be projected into 3D space. Most existing models attempt to solve this by treating objects as independent entities, leading to scenes where furniture might overlap, float, or face nonsensical directions.

This research introduces the Holistic Pose Graph (HPG), a framework that shifts the focus from individual object detection to modeling the global geometric structure of a scene. By treating the scene as a network of interconnected relationships, the model can resolve 2D-to-3D ambiguity through collective reasoning.

The Holistic Pose Graph Framework

The core of the proposed method is the HPG, which represents a scene as a fully connected graph. In this structure, nodes represent individual object poses (including center offset, distance from the camera, orientation, and scale), while edges represent the relative pose between pairs of objects.

Overview of the Holistic Pose Graph (HPG) pipeline, illustrating the integration of camera, object, and relative poses through iterative message passing.

Iterative Message Passing

Rather than relying on a single-shot prediction, the HPG employs Gated Recurrent Units (GRUs) to perform iterative message passing. During this process, nodes update their states based on the information from their neighbors, and edges update based on the nodes they connect. This allows the model to "negotiate" the most physically plausible positions for all objects simultaneously, ensuring that the final 3D scene is globally consistent.

Consistency Loss and Evaluation

A significant innovation in this work is the consistency loss. Mathematically, the relative pose between two objects should equal the difference between their individual 3D poses. The authors enforce this constraint during training, forcing the network's independent object branch and its relational edge branch to stay in sync.

To better measure performance, the researchers also introduced Holistic Pose Estimation (HPE). Unlike standard metrics like 3D Intersection-over-Union (IoU), which only check if a box is in the right place, HPE evaluates the accuracy of triplets (subject-relation-object). This metric more closely aligns with human perception of whether a reconstructed scene is "rational."

Experimental Results

The HPG method was rigorously tested on the SUN RGB-D and NYUv2 datasets, showing substantial improvements over previous state-of-the-art methods like Total3D.

  • Detection Accuracy: On the SUN RGB-D dataset, the model achieved a 32.75% mAP (mean Average Precision), a significant increase from the baseline of 26.38%.
  • Box Estimation: When provided with ground-truth 2D boxes, the 3D mIoU improved from 18.27% to 20.04%.
Qualitative comparison between baseline predictions (a) and the proposed HPG method (b) across office and bedroom scenes.

Qualitative results demonstrate that the HPG method produces much more coherent scenes. As seen in the comparisons above, the holistic approach prevents the "floating" or misaligned boxes common in independent detection methods, resulting in layouts that respect the underlying physics of the environment.

Conclusion and Future Work

By modeling geometry as a constraint rather than a byproduct, the Holistic Pose Graph effectively reduces the search space for 3D positions in monocular images. While the system currently depends on the quality of initial 2D detections and faces quadratic computational growth in very crowded scenes, it represents a major step toward "team-based" object detection. Future work may involve integrating 3D mesh reconstruction and room layout estimation to create even more detailed and immersive 3D scene understandings.

Summary updated

Abstract

Due to the missing depth cues, it is essentially ambiguous to detect 3D objects from a single RGB image. Existing methods predict the 3D pose for each object independently or merely by combining local relationships within limited surroundings, but rarely explore the inherent geometric relationships from a global perspective. To address this issue, we argue that modeling geometric structure among objects in a scene is very crucial, and thus elaborately devise the Holistic Pose Graph (HPG) that explicitly integrates all geometric poses including the object pose treated as nodes and the relative pose treated as edges. The inference of the HPG uses GRU to encode the pose features from their corresponding regions in a single RGB image, and passes messages along the graph structure iteratively to improve the predicted poses. To further enhance the correspondence between the object pose and the relative pose, we propose a novel consistency loss to explicitly measure the deviations between them. Finally, we apply Holistic Pose Estimation (HPE) to jointly evaluate both the independent object pose and the relative pose. Our experiments on the SUN RGB-D dataset demonstrate that the proposed method provides a significant improvement on 3D object prediction.
BibTeX
@inproceedings{xiao2021holistic,
  author = {Jiwei Xiao and Ruiping Wang and Xilin Chen},
  title = {{Holistic Pose Graph: Modeling Geometric Structure among Objects in a Scene using Graph Inference for 3D Object Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=q76aruPL9GU},
}
Figure