Local to Global: Efficient Visual Localization for a Monocular Camera

Sang Jun Lee, Deokhwa Kim, Sung Soo Hwang, Donghwan LeeWACV 2021

TL;DR

This paper introduces a novel real-time monocular visual localization system that combines efficient client-side visual odometry with robust server-side localization. By introducing "SuperKeyFrames" and a map alignment mechanism, it achieves state-of-the-art accuracy comparable to heavier methods while operating in real-time on a single camera, making it suitable for embedded devices.

What Problem Does This Paper Address?

Accurate and robust visual localization (VL) is crucial for applications like autonomous driving and augmented reality. While recent VL algorithms have achieved impressive accuracy, they often suffer from high computational costs, requiring powerful servers and limiting their use on resource-constrained devices like mobile robots or smartphones. Furthermore, many existing methods process each image independently, leading to jerky camera trajectories and poor spatial-temporal consistency, which degrades user experience. This paper aims to bridge this gap by developing a real-time VL system that is both accurate and efficient, capable of running on a monocular camera without additional sensors.

Methods

The proposed system is a hybrid client-server architecture.

  • Client-Side Visual Odometry (VO): This component runs in real-time on the client device. It uses handcrafted features (like ORB) to generate local maps and estimate relative camera movements. This part is designed to be very fast, achieving around 30 frames per second (fps) on a laptop without a GPU.
  • Server-Side Visual Localization (VL): This component leverages powerful learned features (like SuperPoint) for robust localization. It performs more computationally intensive tasks but is optimized to reduce its workload.
  • SuperKeyFrame: This is a novel concept introduced to bridge the gap between the client-side (handcrafted features) and server-side (learned features). A SuperKeyFrame stores both learned features (from the server) and handcrafted features (from the client), along with their 3D structure information. This allows for efficient communication and integration between the two sub-systems.
  • Map Alignment (MA) Sub-system: This crucial component aligns the local maps generated by the VO sub-system to the global coordinates provided by the VL sub-system. It uses a scale factor estimator (employing a Kalman Filter) to handle the inherent scale ambiguity of monocular systems and ensures spatial consistency.
  • Place Recognition using Sequential Information: The system incorporates a probabilistic model for place recognition that leverages sequential image information, making it more robust in scenarios with sparse locations.

Datasets

The paper evaluates its system on two popular datasets:

  • 7-Scenes dataset: This dataset consists of indoor scenes in small-scale environments.
  • Oxford RobotCar dataset: This dataset features outdoor scenes in large-scale environments, including significant illumination and appearance changes.

The reference maps for these datasets are provided with depth information and poses (for 7-Scenes) or GPS and inertial navigation system data (for RobotCar).

Key Insights and Ideas

  • Hybrid Approach is Key: Combining fast, handcrafted features for local tracking (VO) with robust, learned features for global localization (VL) is an effective strategy for achieving both real-time performance and high accuracy.
  • SuperKeyFrames as the Bridge: The concept of SuperKeyFrames is a clever way to unify heterogeneous features from different sub-systems, enabling seamless integration and reducing computational overhead on the server.
  • Map Alignment for Global Consistency: The Map Alignment sub-system, particularly its scale estimation, is critical for transforming local, scale-ambiguous maps into globally consistent, accurate poses.
  • Sequential Information Improves Place Recognition: Leveraging the temporal consistency of camera motion significantly enhances the robustness of place recognition, especially in challenging environments.
  • Real-time Performance on Monocular Input: The system demonstrates that high-accuracy visual localization is achievable in real-time using only a monocular camera, without requiring additional sensors like IMUs or GPS.

Key Results

  • The proposed system achieves accuracy comparable to state-of-the-art structure-based methods like DSAC++ and end-to-end methods like vLocNet.
  • It operates in real-time, achieving up to 25 fps on the client-side (VO sub-system), making it suitable for embedded applications.
  • On the 7-Scenes dataset, the system achieved median translation errors as low as 0.02m and rotation errors of 2.34° (with SuperORB and sequential place recognition).
  • On the challenging Oxford RobotCar dataset, the system achieved median translation errors under 6m and rotation errors under 3m, outperforming other methods significantly.
  • The use of SuperORB (a learned feature detector for ORB) demonstrated improved robustness in scale estimation compared to SuperPoint.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the system is efficient, the VL sub-system still relies on a server for its core localization tasks. The performance of the client-side VO is dependent on the quality of handcrafted features.
  • Future Work: The authors suggest that the proposed system can be integrated into existing SLAM/VO-based systems, implying potential for further enhancements and broader adoption. The success of SuperORB suggests further exploration into learned features that are compatible with traditional methods. The system's ability to operate without extra sensors is a strong foundation for future work on low-cost, robust localization modules.

Summary updated

BibTeX
@inproceedings{lee2021local,
  author = {Sang Jun Lee and Deokhwa Kim and Sung Soo Hwang and Donghwan Lee},
  title = {{Local to Global: Efficient Visual Localization for a Monocular Camera}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure