TL;DR
This paper presents a novel approach for metric localization using Google Street View panoramas as a global map source. By formulating localization as a two-phase non-linear least squares problem, the method estimates 3D feature points from monocular camera sequences and then computes the rigid body transformation between these points and Street View panoramas, achieving submeter accuracy.
What Problem Does This Paper Address?
Accurate metric localization is a fundamental challenge in mobile robotics, enabling applications like autonomous navigation and intelligent driving. Traditional methods often rely on building detailed maps of the environment beforehand, which is time-consuming, expensive, and requires robots to visit the area in advance. This paper addresses the need for a localization method that leverages readily available, large-scale, and globally registered visual data, specifically Google Street View, without the need for prior map construction or robot exploration.
Methods
The core of the proposed method lies in a two-phase non-linear least squares optimization.
- 3D Feature Point Estimation: Given a stream of monocular camera images and odometry estimates, the first phase estimates the 3D positions of tracked feature points. This involves feature detection (SIFT), matching between consecutive frames, and using a homography constraint with RANSAC to filter outliers. The 3D points are then estimated by solving a non-linear least squares problem that incorporates both bearing constraints from image observations and odometry constraints between camera poses. This is done in two steps: first, optimizing 3D points while keeping camera poses fixed, and then performing a full joint optimization of both.
- Metric Localization with Street View Panoramas: The second phase uses the estimated 3D points to find and localize against Google Street View panoramas. A bag-of-words model is used for initial place recognition to find candidate panoramas within a certain radius. Then, feature matching is performed between the monocular images and rectilinear views extracted from these candidate panoramas. Finally, another non-linear least squares problem is formulated to compute the rigid body transformation (6DOF pose) between the Street View panoramas and the estimated 3D points. By leveraging the known GPS coordinates of Street View panoramas, the system can then compute the camera's global GPS position.
Datasets
- Google Street View: Used as the primary source of global, geotagged panoramic imagery.
- Manually Acquired Panoramas: Used in the Freiburg, Germany urban scenario.
- Google Tango Data: Used for urban localization experiments.
- Visual Fiducials (AprilTags): Used in the outdoor parking lot experiment to establish ground truth for panorama positions.
Key Insights and Ideas
- Leveraging Existing Global Data: The paper demonstrates the feasibility of using readily available Street View data as a robust global map for localization, bypassing the need for custom map building.
- Two-Phase Optimization: The separation of 3D point estimation and panorama matching into two distinct optimization phases simplifies the problem and allows for efficient computation.
- Submeter Accuracy with Monocular Input: The method achieves impressive submeter localization accuracy using only a monocular camera and odometry, which are common sensor modalities.
- Robustness to Urban Environments: The experiments show the method's effectiveness in diverse urban settings, including areas with potential GPS signal loss.
Key Results
- Parking Lot Experiment: Achieved submeter accuracy in an outdoor parking lot experiment using AprilTags for ground truth. Approximately 40% of runs were within 1m accuracy, and 60% within 1.5m.
- Urban Localization: Demonstrated successful localization in urban environments using a Google Tango device, even in scenarios with GPS signal loss.
- Comparison to GPS: The paper suggests that the achieved accuracy is significantly better than typical GPS accuracy on mobile devices (5-8.5m).
Limitations and Future Work
- Dependence on Street View Availability: The method's effectiveness is contingent on the availability and quality of Street View imagery in the target environment.
- Odometry Accuracy: While the method is agnostic to odometry type, its performance is influenced by the accuracy of the provided odometry estimates.
- Feature Tracking Limitations: Tracks can be broken due to occlusions or significant viewpoint changes, and the current approach does not attempt to merge broken tracks.
- Future Work: The authors suggest that a single-step optimization could be explored, and the method could be extended for loop closure detection and more accurate reconstruction. They also note that the current implementation works offline, implying online operation as a future direction.
Summary updated