Abstract
Generally capable Spatial AI systems must build persistent scene representations where geometric models are combined with meaningful semantic labels. The many approaches to labelling scenes can be divided into two clear groups: view-based which estimate labels from the input view-wise data and then incrementally fuse them into the scene model as it is built; and map-based which label the generated scene model. However, there has so far been no attempt to quantitatively compare view-based and map-based labelling. Here, we present an experimental framework and comparison which uses real-time height map fusion as an accessible platform for a fair comparison, opening up the route to further systematic research in this area.
Introduction
Spatial AI systems require a dual understanding of their environment: the geometric structure (where things are) and the semantic meaning (what things are). In the context of real-time Simultaneous Localization and Mapping (SLAM), two primary paradigms for semantic labelling have emerged. The view-based approach processes individual camera frames through a neural network and fuses the resulting labels into a global 3D model. Conversely, the map-based approach performs labelling directly on the reconstructed 3D representation.
Historically, the choice between these strategies has been driven by intuition rather than empirical evidence. A significant barrier to comparison has been the use of disparate architectures—typically 2D CNNs for images and 3D volumetric networks for maps. This work introduces a controlled framework to quantitatively evaluate these two approaches using a unified architecture.
Methodology
To facilitate a fair "apples-to-apples" comparison, the study utilizes Height Map Fusion as the mapping backend. By representing the environment as a 2.5D height map, the researchers could employ the same 2D Convolutional Neural Network (CNN) architecture—a modified FuseNet—for both view-based and map-based tasks.
View-Based Labelling
In this configuration, the CNN processes RGB-D frames at their native sensor resolution. The predicted labels are incrementally integrated into the global map using a Bayesian update scheme. This probabilistic fusion allows the system to refine label estimates over time, potentially mitigating transient errors from individual viewpoints.
Map-Based Labelling
Here, the CNN operates on the fused height map. To accommodate scenes of arbitrary size within GPU memory constraints, a sliding window technique is employed. This method ensures that the network maintains sufficient spatial context by using a specific offset during inference.
Experimental Setup
Evaluation was conducted on a custom synthetic dataset designed for high-fidelity ground truth.
- Data Source: 647 synthetic scenes featuring objects from ShapeNet (e.g., keyboards, airplanes) on tabletops.
- Rendering: Photorealistic rendering was achieved using the SceneNet RGB-D methodology.
- Noise Simulation: To assess robustness, Gaussian noise was systematically added to camera poses and depth measurements, simulating real-world sensor imperfections.
Results and Discussion
The experimental results highlight distinct trade-offs between the two labelling paradigms:
- Accuracy and Precision: In noise-free scenarios, the map-based approach achieved superior accuracy, with a mean Intersection over Union (mIoU) of 0.922, compared to 0.889 for the view-based method. Map-based labelling was particularly effective at defining sharp object boundaries.
- Robustness to Pose Noise: The view-based approach demonstrated greater resilience to camera tracking errors. The incremental Bayesian fusion acts as a temporal filter, averaging out misalignments across multiple frames.
- Robustness to Depth Noise: Surprisingly, the map-based method proved more robust to depth noise. The authors suggest that depth noise causes significant 2D segmentation errors in individual views that are difficult for the fusion process to rectify.
- Computational Efficiency: Map-based methods are inherently more efficient. While view-based systems redundantly process overlapping regions across many frames, map-based systems process each part of the scene only once.
Conclusion and Future Work
This study provides the first quantitative evidence that map-based labelling generally offers higher accuracy and efficiency, while view-based fusion provides critical robustness to pose uncertainty. The 2.5D height map representation, while effective for tabletop scenes, remains a limitation for complex 3D environments.
Future research could explore hybrid architectures that leverage the low-latency initial estimates of view-based fusion alongside the high-precision refinement of map-based processing. Extending this framework to full 3D volumetric representations and real-world datasets will be essential for the next generation of Spatial AI systems.
Summary updated