Abstract
Indoor scene analysis is important for some applications, such as augmented reality. Specifically, we confine the indoor scene analysis problem to two aspects: reconstructing the geometry and understanding the observed element. Traditional analysis methods focusing on 2D RGB images are limited because of the lack of stereo measurements, and thus RGBD-based indoor scene analysis has received wide attention. However, geometric analysis and semantic analysis have been treated as two separate problems in most recent works, leaving scene analysis incomplete. In our work, we combine a deep network architecture with a 3D online reconstruction algorithm and propose a complete pipeline to simultaneously analyse the indoor scene from the geometric level and the semantic level. We take a live depth camera as input and consider the scene analysis as two steps. The first step estimates the camera pose and labels scene objects for a single view. The second step fuses the scene objects into an integrated map for a global view. Specifically, we first transfer the input frame to geometric maps and propose a structural constraint iterative closest point (SC-ICP) algorithm for camera tracking. Then, we propose a structural constraint recurrent neural network (SC-RNN) to generate a semantic map for each frame. Finally, the geometric maps and semantic maps from multiple viewpoints can be fused into a complete model according to the camera pose. Our method can improve the accuracy of camera pose estimation and significantly analyse 3D indoor scenes consisting of high-quality geometric details and rich semantic information. It is noteworthy that our method can meet real-time requirements with frame rates of ≈25 Hz.
Introduction
Indoor scene analysis is a cornerstone for applications like augmented reality and robotics, requiring both geometric reconstruction (the "where") and semantic understanding (the "what"). Traditionally, these tasks are treated as separate silos. Geometric tracking often suffers from drift in low-texture environments, while semantic labeling often lacks temporal consistency across frames, leading to "flickering" results. This research proposes a unified pipeline that leverages structural constraints—specifically planar and curved surface primitives—to bridge this gap, achieving real-time performance at approximately 25 Hz.
Proposed Methodology
The proposed system follows a two-stage pipeline: Single View Implementation and Global View Fusion.
1. Structural Primitives Segmentation
The process begins by segmenting the input depth map into two primary categories: planar regions (e.g., walls, floors) using RANSAC and PCA, and curved surfaces (e.g., chairs, mugs) using quadratic surface fitting. These primitives provide the structural context for subsequent tracking and labeling.
2. SC-ICP (Structural Constraint Iterative Closest Point)
Standard ICP algorithms match points based primarily on distance. The proposed SC-ICP improves this by incorporating structural similarity. When searching for corresponding points between frames, it minimizes a weighted sum of distance, normal orientation, and structural type. This significantly reduces pose estimation errors in environments that lack distinct visual textures but possess clear geometric structures.
3. SC-RNN (Structural Constraint Recurrent Neural Network)
For semantic understanding, the system employs an ENet-based encoder-decoder architecture enhanced with a Recurrent Unit (RU). The network utilizes curvature maps as a primary feature, which are more invariant to viewpoint changes than raw depth. Temporal consistency is maintained by using the estimated camera pose to spatially align hidden states between frames, ensuring the model "remembers" object locations accurately as the camera moves.
Key Insights: Curvature and Structure
A critical innovation is the emphasis on curvature as a robust feature for identifying structural boundaries. By projecting previous features into the current coordinate system using the estimated pose, the system achieves spatial-temporal consistency without the heavy computational cost of 3D convolutions.
Experimental Results
The pipeline was evaluated using the ScanNetV2 dataset for semantic training and the TUM RGB-D Benchmark for tracking stability.
- Real-Time Performance: The entire pipeline runs at approximately 25 Hz, making it suitable for live AR applications.
- Segmentation Accuracy: On the ScanNet dataset, the SC-RNN achieved a mean IoU (Intersection over Union) of 0.569, significantly outperforming established methods like PointNet++ (0.339).
- Tracking Precision: SC-ICP showed a significant reduction in Root Mean Squared Error (RMSE) compared to classic point-to-plane ICP, particularly in scenes with complex geometry.
Conclusion
By integrating geometric structure into both tracking and semantic learning, this method provides a robust solution for real-time 3D scene analysis. While the system excels at large structures, future work will focus on improving the detail of small object reconstruction and exploring more flexible geometric representations beyond quadratic surfaces.
Summary updated