The SLAM Hive Benchmarking Suite

Yuan-Yuan Yang, Bowen Xu, Yinjie Li, Sören SchwertfegerICRA 2023 · OpenReview

TL;DR

This paper introduces SLAM Hive, a novel, containerized benchmarking suite designed to systematically evaluate Simultaneous Localization and Mapping (SLAM) algorithms. By leveraging Docker and enabling cluster deployment, SLAM Hive can perform thousands of mapping runs, allowing for unprecedented exploration of the vast parameter space in SLAM systems and driving more robust scientific comparisons.

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) is a fundamental capability for autonomous robots, enabling them to build maps of their environment while simultaneously tracking their own position within that map. However, evaluating SLAM algorithms is a complex challenge. SLAM systems have a multitude of configuration options (e.g., sensor settings, algorithm parameters) and can be influenced by various factors like environment type and robot trajectory. To scientifically assess their performance, experiments need to be reproducible and repeatable. While pre-recorded datasets with ground truth are crucial, the sheer number of possible configurations makes exhaustive testing impractical. Existing benchmarking tools often have limitations in terms of the datasets they support, the algorithms they can evaluate, or their ability to automate large-scale, multi-parameter comparisons. This leaves a significant gap in the ability of researchers and engineers to thoroughly explore and compare SLAM algorithms, hindering scientific progress.

Methods

The core of SLAM Hive is its containerized architecture, utilizing Docker to package SLAM algorithms, evaluation tools, and the entire system. This ensures reproducible environments for each component. The suite is designed for scalability and can be deployed on a single workstation or a cluster. Key components include:

  • Web Controller: A Flask-based graphical user interface (GUI) for users to configure tasks, select algorithms and datasets, and view results.
  • Configuration Generator: Automates the creation of thousands of unique mapping run configurations by permuting algorithms, datasets, and parameters.
  • Mapping Container: Executes SLAM algorithms within Docker containers, mounting necessary scripts, parameters, and datasets. It also monitors CPU and memory usage.
  • Evaluation Container: Runs evaluation algorithms to compare estimated trajectories against ground truth, calculating metrics like Absolute Trajectory Error (ATE) and Relative Pose Error (RPE).
  • Database: Stores all historical task information, including parameters, configurations, and evaluation results, enabling meta-analysis.
  • Meta Analysis Tool: Aggregates and visualizes historical data to identify trends and correlations between parameters and performance.

The system employs a standard interface for algorithms, requiring them to adhere to specific scripting and output formats, making it extensible to new algorithms, even closed-source ones or those not using ROS, as long as they can be integrated into a Docker container.

Datasets

The paper mentions the use of public datasets for evaluation:

  • KITTI dataset: A well-known dataset for autonomous driving scenarios.
  • TUM RGB-D dataset: Provides RGB-D data with depth values.
  • EuRoC MAV dataset: Collected by UAVs, often used for visual-inertial odometry.

The authors also highlight the need for more comprehensive datasets and mention their ongoing work on the "ShanghaiTech Mapping Robot II" to create such datasets in the future.

Key Insights and Ideas

  • Bridging the Parameter Space Gap: SLAM Hive directly addresses the challenge of exploring the vast parameter space of SLAM algorithms, which was previously infeasible due to the computational cost of running thousands of experiments.
  • Containerization for Reproducibility: The use of Docker is crucial for ensuring that experiments are reproducible and that environments are consistent across different runs and systems.
  • Scalability for Large-Scale Evaluation: The ability to deploy SLAM Hive in a cluster allows for the execution of an unprecedented number of mapping runs, enabling more thorough and statistically significant evaluations.
  • Meta-Analysis for Deeper Understanding: The integrated meta-analysis feature allows researchers to move beyond single-run evaluations and identify complex relationships between algorithm parameters, hardware configurations, and performance metrics.
  • Extensibility and Openness: The standardized interface and containerized approach make SLAM Hive highly extensible, encouraging community contributions and the evaluation of a wider range of SLAM systems.

Key Results

The paper presents experimental results demonstrating SLAM Hive's capabilities:

  • Performance Comparison: Experiments on the EuRoC dataset showed that ORB-SLAM3 generally performed more stably and achieved higher accuracy than ORB-SLAM2, though ORB-SLAM3 consumed more memory.
  • Parameter Impact: An analysis of ORB-SLAM2 on the TUM RGB-D dataset revealed that increasing the number of ORB feature points generally decreased ATE and RPE but increased CPU and memory usage. It also highlighted that a low number of features (250) could lead to algorithm failure.
  • Resource Usage: The experiments also provided quantitative data on CPU and memory usage for different algorithms and configurations, illustrating trade-offs between performance and resource consumption. For instance, ORB-SLAM2's stereo mode showed significantly higher CPU usage than its monocular mode.

Limitations and Future Work

  • Automated Configuration Generation: While the system supports automated generation of configurations, the paper notes that this component is still under development.
  • Kubernetes Orchestration: Deployment in a cluster is planned to optionally support Kubernetes for more robust orchestration.
  • Real-time Performance: Future work includes exploring the real-time performance of algorithms and potentially supporting GPU-accelerated systems.
  • User Management: A user management system is planned to allow registered users to add algorithms, datasets, and evaluation containers.
  • New Datasets: The authors emphasize the need for more challenging and diverse datasets, which they plan to address with their own mapping robot.
  • Broader Applications: The authors suggest that SLAM Hive's flexible framework could be adapted for benchmarking other robotics and computer vision tasks, such as sensor calibration, segmentation, and object recognition.

Summary updated

Abstract

Benchmarking Simultaneous Localization and Mapping (SLAM) algorithms is important to scientists and users of robotic systems alike. But through their many configuration options in hardware and software, SLAM systems feature a vast parameter space that scientists up to now were not able to explore. The proposed SLAM Hive Benchmarking Suite is able to analyze SLAM algorithms in 1000's of mapping runs, through its utilization of container technology and deployment in a cluster. This paper presents the architecture and open source implementation of SLAM Hive and compares it to existing efforts on SLAM evaluation. Furthermore, we highlight the function of SLAM Hive by exploring some open source algorithms on public datasets in terms of accuracy. We compare the algorithms against each other and evaluate how parameters effect not only accuracy but also CPU and memory usage. Through this we show that SLAM Hive can become an essential tool for proper comparisons and evaluations of SLAM algorithms and thus drive the scientific development in the research on SLAM.
BibTeX
@inproceedings{yang2023slam,
  author = {Yuan-Yuan Yang and Bowen Xu and Yinjie Li and Sören Schwertfeger},
  title = {{The SLAM Hive Benchmarking Suite}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=poCvcPodzU},
}
Figure