SeDAR - Semantic Detection and Ranging: Humans can localise without LiDAR, can robots?

Oscar Mendez Maldonado, Simon Hadfield, Nicolas Pugeault, Richard Bowden

Abstract

How does a person work out their location using a floorplan? It is probably safe to say that we do not explicitly measure depths to every visible surface and try to match them against different pose estimates in the floorplan. And yet, this is exactly how most robotic scan-matching algorithms operate. Similarly, we do not extrude the 2D geometry present in the floorplan into 3D and try to align it to the real-world. And yet, this is how most vision-based approaches localise. Humans do the exact opposite. Instead of depth, we use high level semantic cues. Instead of extruding the floorplan up into the third dimension, we collapse the 3D world into a 2D representation. Evidence of this is that many of the floorplans we use in everyday life are not accurate, opting instead for high levels of discriminative landmarks. In this work, we use this insight to present a global localisation approach that relies solely on the semantic labels present in the floorplan and extracted from RGB images. While our approach is able to use range measurements if available, we demonstrate that they are unnecessary as we can achieve results comparable to state-of-the-art without them.

Introduction: Localizing Beyond Geometry

Most autonomous robots navigate using "scan-matching," a process that aligns precise distance measurements from LiDAR or RGB-D sensors with a 3D model or an extruded 2D floorplan. While effective, this approach is computationally intensive and brittle. Minor inaccuracies in the floorplan or drift in the robot's scale estimation can lead to total system failure.

In contrast, human navigation relies on high-level semantic landmarks. We don't calculate the exact distance to every wall; we recognize a door to our left or a window ahead and instantly orient ourselves on a 2D map. SeDAR (Semantic Detection and Ranging) aims to bridge this gap by teaching robots to localize using these same semantic cues.

The SeDAR Framework

The core of the SeDAR framework is the "SeDAR Sensor," which transforms standard visual data into a format suitable for 2D localization.

The SeDAR sensor pipeline: (A) RGB input, (B) semantic segmentation, and (C) the resulting 2D semantic scan projected into the environment.

Using a Convolutional Neural Network (CNN) encoder-decoder, the robot labels objects in its RGB camera view. Instead of attempting to reconstruct a full 3D scene, SeDAR "collapses" the 3D information from the center scanline of the image into a 2D semantic "scan." This approach simplifies the localization problem and reduces the sensitivity to environmental variables like ceiling height.

Semantic Mapping and Localization

Traditional occupancy grids are replaced with Semantic Floorplans. These maps explicitly label pixels as "wall," "door," or "window," providing the robot with a rich set of landmarks.

A semantic floorplan where architectural features like walls, doors, and windows are explicitly labeled for robot localization.

To handle real-world map inaccuracies, SeDAR introduces the "Ghost Factor" into its Monte Carlo Localization (MCL) algorithm. Unlike standard MCL, which eliminates particles that intersect walls, the Ghost Factor allows particles some leeway to pass through "soft" obstacles like doors, acknowledging that human-made floorplans are rarely perfect.

Furthermore, SeDAR utilizes Semantic Raycasting. In its "range-less" mode, the system ignores depth entirely, focusing only on the angle at which specific labels appear. This makes the localization invariant to scale errors. The system also employs Adaptive Uncertainty, automatically placing higher trust in rare landmarks (e.g., a single window in a long corridor) than in common ones.

Evaluation and Results

The researchers evaluated SeDAR using a custom TurtleBot dataset recorded in a 49m x 49m office environment, with ground truth provided by RTAB-Map.

  • Accuracy: SeDAR (using range + labels) achieved a Root Mean Square Error (RMSE) of 0.19m, outperforming the standard AMCL (0.24m).
  • LiDAR-Free Performance: Even without depth information, relying solely on the angular distribution of semantic labels from an RGB camera, SeDAR achieved results comparable to traditional LiDAR-based methods.
  • Global Localization: In "lost robot" scenarios, SeDAR converged significantly faster. Combined range and labels reduced global RMSE from 7.31m (AMCL) to 4.78m.
  • Efficiency: The system maintains real-time performance, running at 8–10 frames per second on modern hardware.

Conclusion

SeDAR demonstrates that semantic information is a powerful and robust alternative to pure geometric sensing. By focusing on what things are rather than just where they are, robots can achieve human-like localization performance. Future work will focus on reducing the computational overhead of semantic segmentation and automating the generation of semantic maps from legacy architectural drawings.

Summary updated

Abstract

How does a person work out their location using a floorplan? It is probably safe to say that we do not explicitly measure depths to every visible surface and try to match them against different pose estimates in the floorplan. And yet, this is exactly how most robotic scan-matching algorithms operate. Similarly, we do not extrude the 2D geometry present in the floorplan into 3D and try to align it to the real-world. And yet, this is how most vision-based approaches localise. Humans do the exact opposite. Instead of depth, we use high level semantic cues. Instead of extruding the floorplan up into the third dimension, we collapse the 3D world into a 2D representation. Evidence of this is that many of the floorplans we use in everyday life are not accurate, opting instead for high levels of discriminative landmarks. In this work, we use this insight to present a global localisation approach that relies solely on the semantic labels present in the floorplan and extracted from RGB images. While our approach is able to use range measurements if available, we demonstrate that they are unnecessary as we can achieve results comparable to state-of-the-art without them.
BibTeX
@inproceedings{maldonado2018sedar,
  author = {Oscar Mendez Maldonado and Simon Hadfield and Nicolas Pugeault and Richard Bowden},
  title = {{SeDAR - Semantic Detection and Ranging: Humans can localise without LiDAR, can robots?}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=8-f7DkNjxqY},
}
Figure