Abstract
In this paper, we tackle the problem of indoor scene understanding using RGBD data. Towards this goal, we propose a holistic approach that exploits 2D segmentation, 3D geometry, as well as contextual relations between scenes and objects. Specifically, we extend the CPMC framework to 3D in order to generate candidate cuboids, and develop a conditional random field to integrate information from different sources to classify the cuboids. With this formulation, scene classification and 3D object recognition are coupled and can be jointly solved through probabilistic inference. We test the effectiveness of our approach on the challenging NYU v2 dataset. The experimental results demonstrate that through effective evidence integration and holistic reasoning, our approach achieves substantial improvement over the state-of-the-art.
Introduction
Detecting objects in 3D is a fundamental requirement for indoor robotics, as navigation and manipulation depend on spatial awareness beyond flat pixel coordinates. However, 3D detection is notoriously difficult due to depth ambiguity in monocular images. While RGB-D sensors provide depth maps, converting raw point clouds into recognized objects remains a challenge. Most existing detectors treat objects in isolation, ignoring the physical rules of the world—objects rest on floors, lean against walls, and appear in specific room types. This research aims to integrate these physical and contextual "priors" into the detection process to resolve sensor ambiguities.
Methods
The proposed multi-stage pipeline moves from raw pixels to a structured 3D scene representation.
3D Region Candidates and Cuboid Fitting
The framework extends the Constrained Parametric Min-Cuts (CPMC) framework to 3D. By utilizing depth information, the system identifies "blobs" in the point cloud likely representing individual objects. For each candidate region, a 3D cuboid is fitted. To ensure robustness against sensor noise, the minimal box containing 95% of the points is calculated and constrained to be parallel to the floor.
Holistic CRF Model
The core of the system is a Conditional Random Field (CRF) that jointly decides the scene type (e.g., "bedroom") and labels for all detected cuboids. The model utilizes several scoring factors:
- Appearance & Segmentation: 2D visual features from pixels.
- Geometry: 3D size and shape constraints (e.g., refrigerators are typically tall).
- Semantic Context: Room-object compatibility (e.g., sofas are likely in living rooms).
- Geometric Context: Spatial relationships (e.g., chairs near desks, TVs on cabinets).
Experimental Results
The approach was evaluated on the NYU v2 dataset, comprising 1,449 indoor scenes. The researchers manually cleaned the labels into 21 distinct object categories.
Performance Metrics
Compared to the Deformable Part Model (DPM), this holistic approach achieved over 10x higher precision at similar recall levels. The integrated framework reached an overall F1-score of 36%. Inference is remarkably efficient, taking approximately 15ms per scene after candidate generation.
Key Insights
Ablation studies confirmed that "scene-object" context and "object-object" context (e.g., support relationships) consistently improved accuracy. The results demonstrate that appearance (RGB) and geometry (Depth) are highly complementary, and enforcing physical constraints (e.g., objects cannot penetrate walls) effectively filters out false detections.
Conclusion
This work demonstrates that holistic reasoning—combining visual appearance with 3D geometry and semantic context—significantly improves indoor object detection. While the cuboid representation is a simplification, it provides the necessary spatial occupancy for robotic tasks. Future work may address irregular shapes beyond cuboids and extend the model to temporal video sequences for enhanced consistency.
Summary updated