Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping

Antoni Rosinol, Marcus Abate, Yun Chang, Luca Carlone

TL;DR

This paper introduces Kimera, an open-source C++ library that unifies real-time visual-inertial SLAM with 3D metric-semantic reconstruction. Kimera provides a modular framework for robots to simultaneously understand their environment's geometry and the objects within it, enabling more intelligent perception and navigation.

What Problem Does This Paper Address?

Robots need to understand their surroundings in two key ways: geometrically (where are things?) and semantically (what are those things?). While significant progress has been made in Simultaneous Localization and Mapping (SLAM) for geometric understanding and in deep learning for semantic segmentation of 2D images, these two fields have largely developed in isolation. Existing SLAM systems often struggle to provide rich semantic information, and semantic segmentation methods typically operate on 2D images, lacking 3D context. This gap limits a robot's ability to perform complex tasks that require understanding both the shape and identity of objects in its environment, such as "fetch the red cup from the table." Kimera aims to bridge this gap by integrating state-of-the-art visual-inertial odometry (VIO), pose graph optimization (PGO), 3D mesh reconstruction, and semantic labeling into a single, cohesive, and real-time system.

Methods

Kimera is built around four modular components, allowing for flexibility and the ability to run modules independently:

  • Kimera-VIO: This module provides fast and accurate visual-inertial odometry at the IMU rate. It uses a GTSAM-based approach with IMU preintegration and structureless vision factors, achieving top performance on the EuRoC dataset. It can handle both monocular and stereo camera inputs.
  • Kimera-RPGO: A robust pose graph optimization (PGO) module designed to produce globally consistent trajectories. It incorporates advanced outlier rejection techniques to handle perceptual aliasing and reduce the need for extensive parameter tuning.
  • Kimera-Mesher: This module generates lightweight 3D meshes for fast obstacle avoidance. It can create both per-frame meshes (for low-latency detection) and multi-frame meshes that span the VIO's receding horizon, providing a continuously updated geometric representation of the scene.
  • Kimera-Semantics: This module builds a more accurate, global 3D mesh and semantically annotates it. It leverages 2D semantic segmentation from modern deep learning methods and uses a volumetric approach (similar to Voxblox) combined with bundled raycasting to propagate semantic labels into the 3D mesh.

The library is designed to run in real-time on a CPU and is compatible with ROS, offering debugging and visualization tools.

Datasets

The paper primarily evaluates Kimera using the EuRoC micro aerial vehicle datasets. For semantic reconstruction evaluation, they utilize a photo-realistic Unity-based simulator that provides ground truth for geometry and semantics.

Key Insights and Ideas

  • Modularity is Key: The design of Kimera with four distinct modules is a significant strength, allowing researchers to leverage specific components (e.g., just the VIO or the mesher) or combine them as needed. This flexibility is crucial for diverse research applications.
  • Bridging the 2D-3D Semantic Gap: Kimera's approach to semantic annotation by propagating 2D labels into a 3D volumetric representation is a critical step towards truly understanding scenes in 3D.
  • CPU-First, Real-Time Performance: Achieving real-time metric-semantic SLAM on a CPU is a notable accomplishment, making the system more accessible and deployable on resource-constrained platforms compared to GPU-dependent methods.
  • Robustness through PGO: The Kimera-RPGO module's focus on robust loop closure detection and optimization is vital for preventing catastrophic failures in SLAM systems, especially in environments with repetitive structures.

Key Results

  • State-of-the-Art VIO: Kimera-VIO achieves top performance in terms of Absolute Translation Error (ATE) on the EuRoC dataset, comparable to or better than existing open-source VIO pipelines, even when using stereo cameras against monocular benchmarks.
  • Robust PGO: Kimera-RPGO demonstrates robustness to loop closure parameter tuning, showing consistent performance across different threshold values.
  • Accurate Geometric Reconstruction: The global mesh produced by Kimera-Semantics shows high accuracy, with an average error of 0.35-0.48m on the EuRoC dataset.
  • Fast Meshing: Kimera-Mesher can generate per-frame 3D meshes in under 5ms and multi-frame meshes in 15ms, enabling real-time obstacle avoidance.
  • Semantic Accuracy: In simulation, Kimera-Semantics achieves competitive semantic accuracy (e.g., mIoU of 57.23% with dense stereo, and 80.03% with ground truth depth), with the primary performance drop attributed to the limitations of dense stereo in challenging regions.

Limitations and Future Work

  • Dense Stereo Limitations: The paper notes that dense stereo methods, while crucial for detailed reconstruction, struggle with texture-less regions like walls, leading to increased geometric and semantic errors. This is identified as a key area where performance drops.
  • Semantic Annotation Dependency: The quality of the semantic annotation is directly dependent on the accuracy of the input 2D semantic segmentation.
  • Future Work: The authors express hope that Kimera will serve as a solid foundation for future metric-semantic SLAM and perception research, encouraging researchers to build upon its modularity and performance. Specific future directions are not explicitly detailed beyond this general aspiration.

Summary updated

Abstract

We provide an open-source C++ library for real-time metric-semantic visual-inertial Simultaneous Localization And Mapping (SLAM). The library goes beyond existing visual and visual-inertial SLAM libraries (e.g., ORB-SLAM, VINS-Mono, OKVIS, ROVIO) by enabling mesh reconstruction and semantic labeling in 3D. Kimera is designed with modularity in mind and has four key components: a visual-inertial odometry (VIO) module for fast and accurate state estimation, a robust pose graph optimizer for global trajectory estimation, a lightweight 3D mesher module for fast mesh reconstruction, and a dense 3D metric-semantic reconstruction module. The modules can be run in isolation or in combination, hence Kimera can easily fall back to a state-of-the-art VIO or a full SLAM system. Kimera runs in real-time on a CPU and produces a 3D metric-semantic mesh from semantically labeled images, which can be obtained by modern deep learning methods. We hope that the flexibility, computational efficiency, robustness, and accuracy afforded by Kimera will build a solid basis for future metric-semantic SLAM and perception research, and will allow researchers across multiple areas (e.g., VIO, SLAM, 3D reconstruction, segmentation) to benchmark and prototype their own efforts without having to start from scratch.
BibTeX
@inproceedings{rosinol2020kimera,
  author = {Antoni Rosinol and Marcus Abate and Yun Chang and Luca Carlone},
  title = {{Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=hrdnHnqat9},
}
Figure