TL;DR
This paper introduces a novel closed-loop benchmarking framework for Visual-Inertial SLAM (VI-SLAM) systems, moving beyond traditional open-loop evaluations. By simulating a robot following a commanded trajectory, they demonstrate that both estimation drift and, crucially, visual processing latency significantly impact trajectory tracking accuracy, suggesting a need for VI-SLAM systems to balance these factors.
What Problem Does This Paper Address?
Robots navigating in environments without GPS (like indoors or underground) rely heavily on Visual-Inertial SLAM (VI-SLAM) systems for localization and mapping. While VI-SLAM systems fuse data from cameras and inertial measurement units (IMUs) to estimate a robot's pose, their performance is typically evaluated using "open-loop" analysis. This means the SLAM system's output is compared to ground truth without influencing the robot's actual actions. However, the ultimate goal of VI-SLAM is often closed-loop navigation, where the estimated pose is used by a controller to guide the robot. This paper argues that open-loop evaluation doesn't fully capture how VI-SLAM systems perform in these real-world, closed-loop scenarios, particularly concerning the impact of visual estimation latency on trajectory tracking.
Methods
The authors developed a closed-loop benchmarking simulation using Gazebo and ROS. This framework consists of two main components:
- VI-SLAM System: This module takes simulated stereo camera images and IMU data to generate high-rate pose estimates. The paper integrates five different VI-SLAM systems, representing various approaches:
- MSCKF-VIO + MSF: Tightly-coupled VIO with KLT features and an EKF-based fusion.
- VINS-Fusion: Tightly-coupled VI-SLAM with KLT features and Bundle Adjustment (BA) backend, known for higher latency but loop-closing capabilities.
- SVO + MSF: Loosely-coupled VIO with direct visual odometry, noted for low latency.
- ORB-SLAM + MSF: Feature-based SLAM with BA backend, known for computational cost and higher latency.
- Lazy-GF-ORB-SLAM: A modified ORB-SLAM designed to reduce latency by prioritizing immediate pose estimation computations.
- Controller: This module uses the high-rate pose estimates from the VI-SLAM system to generate actuator commands for the simulated robot. The controller aims to make the robot follow a desired trajectory.
The performance is evaluated by comparing the desired trajectory with the actual trajectory the robot follows, quantified by tracking Root Mean Square Error (RMSE). The simulation allows for controlled testing of different VI-SLAM configurations, IMU types (high-end ADIS16448 and low-end MPU6000), and desired velocities.
Datasets
The paper uses a simulated virtual office world as its primary "dataset." This environment is designed to be representative of real-world scenarios and is compared against the well-known EuRoC MAV dataset for validation of the simulation's applicability. Six different test trajectories were created within the simulated environment, varying in length, complexity, and revisit frequency.
Key Insights and Ideas
- Latency Matters (A Lot): The study strongly suggests that visual processing latency in VI-SLAM systems is a critical factor for accurate closed-loop trajectory tracking, potentially as important as estimation drift. Reduced latency leads to better tracking performance, even if the underlying visual estimation is not perfect.
- Drift vs. Latency Trade-off: There appears to be a trade-off between drift and latency. Systems with lower drift might tolerate higher latency, and systems with lower latency might be more forgiving of higher drift. However, achieving sufficiently low latency seems to be a prerequisite for good performance.
- Closed-Loop is Different: Open-loop metrics (like ATE) don't fully predict closed-loop performance. A system that performs well in open-loop might struggle in closed-loop, and vice-versa.
- Simulation is a Valid Tool: The simulated environment, while not perfectly replicating all aspects of real-world datasets, provides a comparable and sometimes harder benchmarking ground, making it a valuable tool for reproducible research.
- GF-ORB-SLAM's Promise: The modified ORB-SLAM (Lazy-GF-ORB-SLAM) shows significant improvements in success rate and reduced RMSE compared to the original ORB-SLAM, attributed to its latency reduction techniques.
Key Results
- GF-ORB-SLAM Outperforms: The Lazy-GF-ORB-SLAM system consistently achieved the lowest tracking RMSE across various configurations, demonstrating superior closed-loop trajectory tracking performance.
- SVO's Mixed Performance: SVO showed good closed-loop performance but poorer open-loop results, highlighting the difference in evaluation methodologies.
- VINS and ORB Failures: Both VINS and ORB exhibited significant failures (navigation errors) in multiple configurations, especially with the low-end IMU.
- Latency Impact: Tables IV and V clearly show that systems with lower average latency (e.g., SVO, GF) generally performed better in closed-loop tracking, especially when compared to systems with higher latency (e.g., VINS, ORB). For instance, GF's latency was significantly lower than ORB's, correlating with its better tracking performance.
- IMU Quality Matters: The performance of some systems, particularly MSCKF, was significantly affected by the quality of the IMU data, indicating an over-reliance on IMU for pose estimation in certain scenarios.
Limitations and Future Work
- Simulation Limitations: While validated, the simulation doesn't perfectly span the entire domain of real-world scenarios. The authors note that future work should include more diverse environments, mobile robots, and rendering options.
- Collision Avoidance: The current simulation excludes collision avoidance to isolate the impact of VI-SLAM. Future work could integrate this to increase task realism.
- Environmental Obstacles: The impact of environmental obstacles on SLAM performance in closed-loop scenarios is an area for future exploration.
- Outlier Cases for GF: While GF performed best overall, the authors mention that some outlier cases still need to be resolved.
- Further Benchmarking: The authors advocate for continued development of open-source, reproducible benchmarking frameworks to guide future VI-SLAM research.
Summary updated