Improved vision-only localization method for mobile robots in indoor environments

Gang Huang, Liangzhu Lu, Yifan Zhang, Gangfu Cao, Zhe ZhouAutonomous 2024 · DOI

TL;DR

This paper introduces a robust vision-only localization system for indoor mobile robots that fuses handcrafted visual features with deep-learning-based semantic recognition. By combining a modified Bag-of-Words (BoW) model with 3D metric calculations, the system achieves over 90% scene recognition accuracy and sub-meter localization precision without the need for expensive LiDAR or restrictive artificial markers like QR codes.

What Problem Does This Paper Address?

For mobile robots in "Sustainable Smart Manufacturing" environments (like warehouses or factories), reaching a target area is only half the battle. To perform precise tasks like docking or picking up items, the robot needs to know its exact pose (position and orientation).

Traditional solutions often rely on expensive LiDAR sensors or "structured" environments filled with artificial markers like QR codes, which limit the robot's flexibility. Purely visual methods often struggle in "unstructured" indoor spaces where lighting changes or repetitive textures can confuse standard algorithms. The authors aim to bridge this gap by creating a vision-only system that is both computationally efficient and robust enough for real-world, unstructured indoor scenes.

Methods

The authors propose a two-stage "offline modeling, online localization" pipeline:

  1. Offline Scene Modeling: The system creates a map by collecting images and 3D data of the environment. It extracts two types of information:
  2. Handcrafted Features: Using the BEBLID (Boosted Efficient Binary Local Image Descriptor) algorithm to find distinct texture points.
  3. Semantic Features: Using YOLOv10 to identify significant indoor objects (e.g., exit signs, elevators, fire hydrants).
  4. Modified BoW Model: They create a hybrid feature vector that combines traditional visual "words" with a frequency-based semantic representation.
  5. Online Localization:
    • Scene Recognition: When the robot sees a new image, it uses a "satisfaction function" to calculate similarity. This function fuses semantic similarity (via cosine distance) and BoW similarity (via Euclidean distance).
  6. ROI-Based Matching: To save time and ignore dynamic noise (like moving people), the system only looks for matching feature points within the "Region of Interest" (ROI) defined by the detected semantic objects.
  7. Pose Estimation: By matching 2D image points to stored 3D coordinates, the system calculates a homography matrix, which is then decomposed to retrieve the robot's precise rotation and translation.

Datasets

The researchers evaluated their system in three distinct real-world scenarios at Wuhan University of Science and Technology:

  • Scenario #1 (Office Building): 3200 $m^2$, 25 nodes, 375 test images.
  • Scenario #2 (Teaching Building): 3500 $m^2$, 15 nodes, 150 test images.
  • Scenario #3 (Underground Parking Lot): 3500 $m^2$, 10 nodes, 100 test images.

The performance was benchmarked against traditional ORB-based methods, standard BoW, and a non-visual WiFi fingerprinting method.

Key Insights and Ideas

  • The Hybrid Advantage: Handcrafted features are great for local geometry, but deep learning provides the "big picture" context. By fusing them, the system avoids the "perceptual aliasing" problem (where two different hallways look identical to a computer).
  • Efficiency through Semantics: Using YOLOv10 bounding boxes as ROIs for feature matching is a major "aha moment." It filters out irrelevant background noise and significantly reduces the number of points the system has to process, making it viable for edge hardware like the NVIDIA Jetson Xavier NX.
  • Metric Accuracy without LiDAR: By manually registering 3D coordinates of planar markers (like signs) during the offline phase, they achieve "metric" localization (actual meters/centimeters) using only a monocular camera.

Key Results

  • Recognition Accuracy: The proposed method achieved 94.4% to 98% accuracy across the three scenarios, significantly outperforming standard ORB (approx. 73-79%) and BoW (approx. 85-91%).
  • Localization Precision: The average localization error was consistently low: 0.59m in the office, 0.61m in the teaching building, and 0.64m in the parking lot.
  • Speed: The entire recognition process takes roughly 31.43ms, and the total localization pipeline takes about 98-103ms, making it suitable for real-time robot navigation.
  • Comparison: It vastly outperformed WiFi-based localization, which had average errors exceeding 3 meters and high standard deviations.

Limitations and Future Work

  • Scalability: The current search mechanism for matching query images to map nodes is $O(n)$, meaning computation time increases linearly with the size of the map. For massive industrial complexes, this would need a more hierarchical or topological mapping approach.
  • Static Map Assumption: The system relies on an offline model. Significant changes to the environment (e.g., moving furniture or new signage) would require re-mapping.
  • Future Directions: The authors suggest incorporating multi-frame matching (using a sequence of images rather than just one) and topological maps to improve robustness in even larger, more complex environments.

Summary updated

Abstract

Abstract To solve the problem of mobile robots needing to adjust their pose for accurate operation after reaching the target point in the indoor environment, a localization method based on scene modeling and recognition has been designed. Firstly, the offline scene model is created by both handcrafted feature and semantic feature. Then, the scene recognition and location calculation are performed online based on the offline scene model. To improve the accuracy of recognition and location calculation, this paper proposes a method that integrates both semantic features matching and handcrafted features matching. Based on the results of scene recognition, the accurate location is obtained through metric calculation with 3D information. The experimental results show that the accuracy of scene recognition is over 90%, and the average localization error is less than 1 meter. Experimental results demonstrate that the localization has a better performance after using the proposed improved method.
BibTeX
@inproceedings{huang2024improved,
  author = {Gang Huang and Liangzhu Lu and Yifan Zhang and Gangfu Cao and Zhe Zhou},
  title = {{Improved vision-only localization method for mobile robots in indoor environments}},
  booktitle = {Autonomous},
  year = {2024},
  doi = {10.1007/s43684-024-00075-9},
}
Figure