RGB-D Object SLAM Using Quadrics for Indoor Environments

Ziwei Liao, Wei Wang, Xianyu Qi, Xiaoyu ZhangSensors 2020 · DOI

TL;DR

This paper introduces an object-centric SLAM system for indoor environments that uses quadric surfaces (ellipsoids) to represent objects, enabling a richer semantic understanding for robots. By leveraging RGB-D data and a novel nonparametric pose graph for robust data association, their system significantly improves both localization accuracy and semantic mapping compared to existing methods.

What Problem Does This Paper Address?

Service robots operating in human environments (homes, offices) need to understand and interact with objects based on semantic instructions like "Move to the TV" or "Hand over the cup." Traditional SLAM systems, which rely on geometric features like points, lines, and planes, lack this semantic understanding, making it difficult for robots to follow such commands. While object-centric SLAM aims to solve this, existing methods often struggle with robustly modeling object shapes and orientations, especially in challenging real-world conditions with occlusions and limited viewpoints. Furthermore, accurately associating detected objects across frames (data association) remains a significant challenge, particularly when multiple objects of the same type are present.

Methods

The core of this work is an object-level semantic SLAM system that uses quadric surfaces (specifically ellipsoids) to model objects. This allows for a compact representation of an object's position, orientation, and occupied space.

Key innovations include:

  • RGB-D Camera-Quadric Observation Models: They propose two models to leverage RGB-D data:
  • Complete Observation Model: Uses object detection and point cloud data to estimate a full ellipsoid from a single RGB-D frame. This is enabled by using supporting plane information and object orientation estimation based on dominant surface directions.
  • Partial Observation Model: Activated when depth data is poor (due to occlusion or illumination). It uses bounding boxes from object detection to constrain the ellipsoid.
  • Flexible Switching: A novel evaluation function determines when to switch between the complete and partial models to maximize information utilization.
  • Nonparametric Pose Graph for Data Association: To address the challenging data association problem in object SLAM, they adapt a nonparametric pose graph approach. This method treats data associations as unknown variables and jointly optimizes them with camera poses and object parameters, leading to more robust and globally consistent solutions.
  • Object Orientation Estimation: A novel method estimates object orientation by leveraging the relationship between objects and their supporting planes, and by analyzing the histogram of normal vectors on the object's surface. This is particularly useful for objects with dominant directional features.

Datasets

The authors evaluated their algorithm on:

  • ICL-NUIM Dataset: A virtual dataset providing RGB-D data for home and office scenes.
  • TUM-RGB-D Dataset: A real-world dataset with handheld camera trajectories, covering various desktop and low-texture scenes.
  • Author-Collected Real Mobile Robot Dataset: Recorded in a home-like environment using a Turtlebot3 robot with a Kinect2, featuring more challenging motion patterns and limited viewing angles relevant to mobile robot navigation.

Key Insights and Ideas

  • Quadrics are powerful for object representation: Ellipsoids, as a type of quadric, provide a compact yet informative way to represent an object's 3D extent and orientation, going beyond simple point-based representations.
  • RGB-D data is crucial for robust object SLAM: The availability of depth information significantly reduces the need for multiple observations and large viewpoint changes, which are limitations of monocular quadric SLAM.
  • Flexible observation models are key: The ability to switch between complete and partial ellipsoid models based on data quality is essential for handling real-world challenges like occlusions and poor depth sensing.
  • Nonparametric pose graphs are vital for semantic data association: Treating data associations as unknown variables in a nonparametric pose graph framework is a more effective way to handle the complexities of object SLAM compared to traditional methods.
  • Object semantics can enhance geometric constraints: Incorporating semantic labels to activate or deactivate orientation constraints for specific object types (e.g., only for objects with dominant directions) improves mapping accuracy.

Key Results

  • Improved Localization Accuracy: The proposed algorithm achieved noticeable improvements in trajectory accuracy across all tested datasets, outperforming ORB-SLAM2, NP, and QuadricSLAM. For instance, on the TUM-RGB-D dataset, it showed improvements of up to 53% and 32% in low-texture scenes compared to ORB-SLAM2.
  • Enhanced Mapping Quality: The system demonstrated superior mapping performance, particularly in terms of object translation, rotation, and shape metrics. On the real mobile robot dataset, it achieved significant improvements in translation, rotation, and shape by 41%, 94%, and 63% respectively, compared to QuadricSLAM.
  • Robust Data Association: The nonparametric pose graph approach led to better precision and recall in object instantiation, with the real-robot dataset achieving 91% precision and 1.0 recall.
  • Efficient Initialization: The complete observation model allows for object initialization with a single RGB-D frame, enabling faster map building.

Limitations and Future Work

  • Object Detection Errors: The reliance on YOLOv3, which was not specifically trained for indoor environments, led to some misclassifications and confusion, particularly with similar-looking objects or when objects were viewed from unusual angles. This impacted precision.
  • Point Cloud Segmentation Issues: For objects with very thin structures (like potted plants) or when supporting plane assumptions were incorrect, point cloud segmentation could be inaccurate, leading to errors in the complete constraint model.
  • Object-Support Relationship: The current method for judging object-support relationships could be improved.
  • Object-Object Relationships: Incorporating knowledge about object interactions (e.g., "objects cannot be overlapped") could further enhance data association robustness.

Future work suggested includes:

  • Developing more robust modules for identifying object-support relationships.
  • Introducing object-object relationship constraints for better data association.
  • Training object detectors on indoor scene datasets to improve accuracy and robustness.
  • Exploring semantic navigation based on the generated object-level maps.

Summary updated

Abstract

Indoor service robots need to build an object-centric semantic map to understand and execute human instructions. Conventional visual simultaneous localization and mapping (SLAM) systems build a map using geometric features such as points, lines, and planes as landmarks. However, they lack a semantic understanding of the environment. This paper proposes an object-level semantic SLAM algorithm based on RGB-D data, which uses a quadric surface as an object model to compactly represent the object's position, orientation, and shape. This paper proposes and derives two types of RGB-D camera-quadric observation models: a complete model and a partial model. The complete model combines object detection and point cloud data to estimate a complete ellipsoid in a single RGB-D frame. The partial model is activated when the depth data is severely missing because of illuminations or occlusions, which uses bounding boxes from object detection to constrain objects. Compared with the state-of-the-art quadric SLAM algorithms that use a monocular observation model, the RGB-D observation model reduces the requirements of the observation number and viewing angle changes, which helps improve the accuracy and robustness. This paper introduces a nonparametric pose graph to solve data associations in the back end, and innovatively applies it to the quadric surface model. We thoroughly evaluated the algorithm on two public datasets and an author-collected mobile robot dataset in a home-like environment. We obtained obvious improvements on the localization accuracy and mapping effects compared with two state-of-the-art object SLAM algorithms.
BibTeX
@inproceedings{liao2020rgb,
  author = {Ziwei Liao and Wei Wang and Xianyu Qi and Xiaoyu Zhang},
  title = {{RGB-D Object SLAM Using Quadrics for Indoor Environments}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20185150},
}
Figure