Real-time geometric registration using feature landmark database for augmented reality applications

Takafumi Taketomi, Tomokazu Sato, Naokazu YokoyaProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2009 · DOI

TL;DR

This paper presents a real-time, initialization-free method for estimating a camera's position and orientation (geometric registration) in augmented reality (AR) applications. It achieves this by optimizing a feature landmark database approach to significantly reduce computational costs and eliminate the need for manual initial camera parameter input.

What Problem Does This Paper Address?

Augmented reality (AR) applications rely heavily on accurately aligning virtual objects with the real world. This requires solving the "geometric registration" problem, which means precisely determining the camera's position and orientation in 3D space. While existing methods using feature landmark databases can work in complex environments, they suffer from high computational costs, making them too slow for real-time AR, and require manual input for the initial camera pose. This paper tackles these limitations to enable robust and efficient real-time AR.

Methods

The core of the proposed method is an enhanced feature landmark database approach. The authors introduce two key innovations:

  • Reduced Computational Cost:
    • Tentative Camera Parameter Estimation: Instead of searching the entire database for potential landmarks, the system first estimates tentative camera parameters by tracking landmarks from the previous frame. This narrows down the search space for matching.
  • Landmark Prioritization: Landmarks are assigned "priorities" based on their historical reliability (how often they've been successfully used in previous estimations). This allows the system to focus on the most confident landmarks, further reducing the number of candidates to test.
  • Manual-Initialization Free:
    • Voting Scheme for Initial Parameters: For the very first frame, where no prior information exists, the system uses a voting scheme among potential matching candidates to determine an initial camera pose.

The overall process involves an offline database construction phase (3D reconstruction and landmark feature extraction) and an online camera parameter estimation phase that iteratively refines the camera pose using the prioritized landmarks and tentative estimations.

Datasets

The paper describes constructing a landmark database for an outdoor environment using an omni-directional multi-camera system (Point Grey Research Ladybug). They captured sampled images for database construction and then used SFM (Structure-from-Motion) to obtain 3D positions of image features and camera parameters. For evaluation, they captured another video sequence (720x480 pixels, 15fps, 1,000 frames) and used three training videos to compute landmark priorities.

Key Insights and Ideas

  • Prioritization is Key for Efficiency: Associating reliability scores (priorities) with landmarks is crucial for pruning the search space and achieving real-time performance. This is a significant improvement over methods that treat all landmarks equally.
  • Iterative Refinement for Robustness: The combination of tentative parameter estimation and landmark prioritization creates a feedback loop that progressively refines the camera pose and landmark reliability, leading to more accurate and stable results.
  • Eliminating Manual Input is Practical: The automatic initialization method removes a significant barrier to deploying AR systems in new environments, making them more user-friendly and adaptable.
  • Outdoor Environments are Challenging but Achievable: The paper demonstrates successful application in natural outdoor settings, which are notoriously difficult due to varying illumination and complex geometry.

Key Results

  • Computational Cost Reduction: The proposed method achieved a total computational cost that was approximately 6 times cheaper than the previous method (Method A) and about 21 times cheaper in the matching process (Process B-4) compared to the ideal theoretical reduction. This allows for real-time operation.
  • Accuracy Maintained: Despite the significant speedup, the proposed methods (B, C, and D) did not increase estimation error compared to the previous method (A). In fact, the average re-projection error was slightly reduced in Method D (1.8 pixels) compared to Method A (2.5 pixels).
  • Robustness to Landmark Count: Methods using landmark prioritization (C and D) showed significantly fewer failure frames, especially when the number of available landmarks was lower, compared to methods that did not use prioritization (A and B).

Limitations and Future Work

  • Initial Camera Parameter Estimation Time: The initial camera parameter estimation process for the first frame still takes a considerable amount of time (about 60 seconds), requiring the camera to be fixed during this period.
  • Training Data for Priorities: The paper mentions that system administrators give "several training videos" to determine initial landmark priorities. The process of acquiring and using this training data could be further automated or optimized.

Future work could focus on reducing the initial camera parameter estimation time and exploring more automated methods for learning landmark priorities.

Summary updated

Abstract

In the field of augmented reality, it is important to solve a geometric registration problem between real and virtual worlds. To solve this problem, many kinds of image based online camera parameter estimation methods have been proposed. As one of these methods, we have been proposed a feature landmark based camera parameter estimation method. In this method, extrinsic camera parameters are estimated from corresponding landmarks and image features. Although the method can work in large and complex environments, our previous method cannot work in real-time due to high computational cost in matching process. Additionally, initial camera parameters for the first frame must be given manually. In this study, we realize real-time and manual-initialization free camera parameter estimation based on feature landmark database. To reduce the computational cost of the matching process, the number of matching candidates is reduced by using priorities of landmarks that are determined from previously captured video sequences. Initial camera parameter for the first frame is determined by a voting scheme for the target space using matching candidates. To demonstrate the effectiveness of the proposed method, applications of landmark based real-time camera parameter estimation are demonstrated in outdoor environments.
BibTeX
@inproceedings{taketomi2009real,
  author = {Takafumi Taketomi and Tomokazu Sato and Naokazu Yokoya},
  title = {{Real-time geometric registration using feature landmark database for augmented reality applications}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2009},
  doi = {10.1117/12.805748},
}
Figure