TL;DR
This paper compares the accuracy of three different Simultaneous Localization and Mapping (SLAM)-based indoor mapping technologies: a commercial Matterport system, a high-accuracy LiDAR-based SLAMMER, and a low-cost LiDAR-based NAVIS. The study reveals that while all systems can achieve centimeter-level accuracy, LiDAR-based solutions generally outperform depth-camera-based systems, with SLAMMER demonstrating the best performance due to its high-accuracy LiDAR sensor.
What Problem Does This Paper Address?
The increasing demand for indoor Location-Based Services (LBS) has driven the need for accurate and efficient indoor mapping technologies. Traditional surveying methods are often too slow and labor-intensive for complex indoor environments. SLAM-based systems, which combine localization and mapping in real-time, offer a promising alternative. However, the market offers various SLAM implementations with different sensors (LiDAR, depth cameras, etc.), making it challenging for users to select the most appropriate system based on their budget and accuracy requirements. This paper aims to address this gap by providing a comparative evaluation of three distinct SLAM-based indoor mapping technologies, offering insights into their performance and applicability for different use cases.
Methods
The study compares three SLAM-based indoor mapping systems:
- Matterport: A commercial system using depth cameras (similar to Microsoft Kinect) to capture spatial data and create 3D models. Its 2D map generation mechanism differs from LiDAR-based systems.
- SLAMMER: A high-accuracy, small-footprint LiDAR-based system utilizing a Faro Focus3D 120S LiDAR and an open-source Hector-SLAM algorithm combined with a graph-SLAM backend.
- NAVIS: A low-cost, large-footprint LiDAR-based system developed by FGI, employing a Hokuyo UTM-30LX-EW laser scanner and an Improved Maximum Likelihood Estimation (IMLE) algorithm.
The evaluation involved two distinct indoor scenes: an L-shaped corridor and an open-style library. Accuracy was assessed by comparing the generated maps against reference maps created using a Terrestrial Laser Scanner (TLS). Two feature point selection methods were used: interactive selection and Minimum Bounding Rectangles (MBRs). Registration was performed to align the coordinate systems of the different mapping outputs before accuracy evaluation using Root Mean Square (RMS) error.
Datasets
The study utilized two indoor environments for data collection and evaluation:
- L-shaped corridor: Located on the 2nd floor of the FGI main building, approximately 80 meters long. This scene was primarily used to test the Matterport system and investigate the influence of mapping trajectory and scan settings.
- Open-style library: Located on the 2nd floor of the FGI building, approximately 13x30 meters. This scene was used to compare the accuracy of all three mapping systems (Matterport, SLAMMER, and NAVIS) against a TLS reference.
Reference maps for both scenes were generated using a Faro Focus3D 330X TLS, employing spherical registration targets for accurate alignment.
Key Insights and Ideas
- LiDAR vs. Depth Cameras: LiDAR-based SLAM systems (SLAMMER and NAVIS) generally produce denser and more accurate point clouds than depth-camera-based systems like Matterport. This is attributed to the higher ranging accuracy and smaller footprint of LiDAR sensors.
- Impact of Sensor Footprint and Accuracy: The study highlights that increasing the range accuracy of the LiDAR sensor and decreasing its footprint size can significantly improve indoor mapping accuracy. SLAMMER, with its high-accuracy, small-footprint LiDAR, outperformed NAVIS, which uses a lower-accuracy, larger-footprint LiDAR.
- Trajectory Matters for Matterport: For the Matterport system, the mapping trajectory significantly influences the final mapping results. Different scanning paths and distances between scans can lead to variations in bias error, heading estimation error, and distance error.
- Environmental Challenges for Matterport: The Matterport system can be affected by environmental factors such as reflections from glass surfaces and low illumination, which can bias heading estimation and lead to undetected feature points.
- MBR vs. Interactive Selection: While MBRs offer a more automated approach to feature point extraction, interactive selection proved more effective for the sparser point clouds generated by Matterport, though it can introduce operator bias.
Key Results
- Overall Accuracy: All three SLAM-based systems demonstrated the capability to generate centimeter-level indoor maps.
- SLAMMER Performance: SLAMMER achieved the lowest RMS error (0.020 m for interactive selection, 0.017 m for MBR selection) in the library scene, indicating the highest mapping accuracy among the three.
- NAVIS Performance: NAVIS showed reasonable accuracy, with RMS errors of 0.037 m (interactive) and 0.032 m (MBR).
- Matterport Performance: Matterport exhibited slightly lower accuracy, with RMS errors of 0.042 m (interactive) and 0.047 m (MBR). Its detection rates were also lower, particularly when using the MBR method.
- Point Cloud Density: Matterport's point cloud density was significantly lower (orders of magnitude) compared to SLAMMER and NAVIS.
- Matterport Trajectory Influence: In the L-shaped corridor test, different trajectories for Matterport resulted in varying bias and distance errors, confirming the trajectory's impact on the final map. For instance, a linear trajectory with a 2m scan distance resulted in a bias error of 0.08m and a distance error of -1.25m, while a matrix trajectory with 0.5-3m scan distances yielded a bias error of -1.23m and a distance error of -1.36m.
- Optimal Matterport Setting: For corridor mapping with Matterport, a scan distance of 1 meter between neighboring scans was found to be optimal for balancing accuracy and efficiency.
Limitations and Future Work
- Matterport's Manual Filtering: The paper notes that for Matterport, extra interactive examination might be necessary to rectify undetected results and produce a credible 2D map, implying a need for post-processing.
- Environmental Factors: The study identified challenges for Matterport due to reflections from glass and low illumination, suggesting that future work could explore methods to mitigate these issues.
- Sensor Limitations: The paper acknowledges that the range limit and sensor resolution of Matterport's terrestrial scan mechanism contribute to its lower point cloud density.
- Future Work Suggestions: The authors suggest that further research could explore the impact of different sensor configurations and algorithms on mapping accuracy. They also point to a broad selection of sensors on the market with moderate prices that could be investigated. The paper also implies that further optimization of parameters for different SLAM sensors could be beneficial.
Summary updated