Fast Scene Recognition and Camera Relocalisation for Wide Area Augmented Reality Systems

Tao Guan, Liya Duan, Yongjian Chen, Junqing Yu2010 · DOI

TL;DR

This paper tackles the challenges of real-time scene recognition and camera relocalization for wide-area augmented reality (AR) systems. They introduce adaptive random trees for efficient scene learning and an enhanced PROSAC algorithm for robust camera relocalization, achieving significant improvements in speed and scalability.

What Problem Does This Paper Address?

Augmented reality (AR) aims to overlay virtual objects onto real-world video, creating a seamless experience. A critical challenge for AR, especially in wide-area, unprepared environments, is "registration" – ensuring virtual objects remain perfectly aligned with the real world as the user moves. This requires accurately tracking the camera's position and orientation in real-time. Existing methods often struggle with large-scale environments, dynamic scenes, and recovering from tracking failures. Specifically, online scene learning (recognizing where the camera is) and fast camera relocalization (recovering the camera's pose after losing track) are bottlenecks that limit the performance and usability of wide-area AR systems.

Methods

The authors propose a multi-pronged approach to address these challenges:

  • Adaptive Random Trees for Scene Learning: To overcome the limitations of traditional random trees in handling large maps and numerous scenes, they introduce an adaptive approach. This method dynamically adjusts the classification trees based on the input data, using HSV color values for more robust feature representation and employing Graphics Processing Units (GPUs) to accelerate the recognition process. This leads to better scalability and accuracy compared to previous methods.
  • Enhanced PROSAC for Camera Relocalization: For fast and robust camera relocalization, especially after tracking failures, they enhance the PROSAC algorithm. This involves using a fast and compact natural feature descriptor (from Calonder et al.) and an optimized outlier removal process. The enhanced PROSAC algorithm prioritizes matches with higher similarity scores, significantly reducing computation complexity and enabling rapid relocalization even in maps with thousands of features.
  • Multithreaded Architecture: To achieve real-time performance, the system is implemented using a multithreaded approach. Key tasks like camera tracking, scene mapping, scene learning, and relocalization are separated into four parallel threads running on a multi-CPU architecture. This allows for simultaneous processing and improves overall system responsiveness, enabling the tracking of multiple maps concurrently.

Datasets

The paper mentions using the UKBench image database for evaluating the adaptive random trees. This database contains 2,550 classes, with each class having four images of the same scene. They also describe experiments using custom-built indoor and outdoor maps around their laboratory and campus, with varying numbers of keyframes and 3D points.

Key Insights and Ideas

  • Adaptive Trees for Scalability: The key insight here is that static tree structures struggle with dynamic, large-scale environments. By making the random trees adaptive, they can better partition the feature space and scale to a larger number of scenes and features.
  • HSV for Robustness: Using HSV color space over grayscale or RGB provides a more discriminative and illumination-invariant representation of local patches, crucial for reliable scene recognition.
  • PROSAC's Prioritization: The core idea of PROSAC is that not all matches are equal. By prioritizing matches with higher similarity scores, they can find a correct camera pose much faster, which is critical for real-time AR.
  • Multithreading for Parallelism: Separating computationally intensive tasks into parallel threads is a standard but effective way to achieve real-time performance in complex systems like AR. This allows different parts of the system to work concurrently, preventing bottlenecks.

Key Results

  • Scene Recognition: The adaptive random trees with HSV vectors showed improved recognition rates compared to their previous method, especially with a large number of classes. GPU acceleration significantly reduced recognition time, making it suitable for online implementation.
  • Camera Relocalization: The enhanced PROSAC algorithm demonstrated significantly better performance than traditional RANSAC and other methods in terms of successful relocalization times, especially in scenarios with a high proportion of outliers. For example, in one experiment, their method achieved 417 successful relocalizations compared to 302 for PROSAC and 54 for standard RANSAC.
  • Tracking Performance: The multithreaded system achieved real-time performance, with frame rates of 16.7 fps for single-map tracking, 15.3 fps for two maps, and 12.3 fps for three maps. The system can track multiple maps simultaneously.
  • Feature Descriptors: Ferns-based descriptors were shown to match as many or more points than U-SURF descriptors, with significantly faster computation and matching times.

Limitations and Future Work

  • Memory Usage of Adaptive Random Trees: The paper acknowledges that adaptive random trees can consume significant memory (around 96 MB for the classifier in one experiment), making them potentially impractical for low-memory mobile devices.
  • Tracking Multiple Maps: While the system can track multiple maps simultaneously, it currently only expands a single map at a time. Tracking failures in other maps can occur if the camera moves out of their scope.
  • Future Work: The authors plan to design a flexible compressing algorithm to reduce the memory footprint of adaptive random trees for mobile implementation. They also suggest that merging tracked maps into a single larger map could be an effective future direction to address the limitations of tracking multiple maps independently.

Summary updated

BibTeX
@misc{guan2010fast,
  author = {Tao Guan and Liya Duan and Yongjian Chen and Junqing Yu},
  title = {{Fast Scene Recognition and Camera Relocalisation for Wide Area Augmented Reality Systems}},
  year = {2010},
  doi = {10.3390/s100606017},
}
Figure