A vision-based AR registration method utilizing edges and vertices of 3D model

Ryo Hirose, Hideo SaitoICAT 2005 · DOI

TL;DR

This paper presents a hybrid AR registration method that combines edge and vertex detection from 3D models with auxiliary sensors (magnetic or vision markers) for improved accuracy and robustness. By intelligently handling edge ambiguities and leveraging multiple feature types, the method achieves stable superimposition of virtual objects onto real-world scenes, even with rapid camera movements.

What Problem Does This Paper Address?

Augmented Reality (AR) systems rely heavily on accurate geometric registration to seamlessly blend virtual objects with the real world. This means precisely determining the camera's position and orientation relative to the scene. Traditional methods face challenges: sensor-based approaches (like magnetic sensors or GPS) offer robustness but lack precision, while vision-based methods using artificial markers require extensive setup and limit user movement. Natural feature-based methods are more flexible but struggle with accuracy, especially during rapid camera motion or when dealing with ambiguous features like edges. This paper tackles the problem of achieving both high accuracy and robustness in AR registration, particularly in dynamic environments with challenging visual conditions.

Methods

The proposed method is a hybrid approach that leverages a 3D model of the target object. Here's a breakdown of its key components:

  • Vision-Based Registration with Natural Features: The core of the method uses edges and vertices of the 3D model as natural features.
  • Projection of 3D Model: The 3D model is projected onto the input image plane using predicted camera pose (from the previous frame).
  • Vertex Matching: For each projected vertex, a "search window" is defined. Within this window, feature points are detected using the KLT-Tracker. To refine vertex correspondence, Hough Transform is used to detect lines, and the intersection of these lines is considered as a potential vertex. The combination of KLT features and line intersections helps in robustly identifying vertex correspondences.
  • Edge Matching: This is a crucial innovation. Instead of relying on a single point with the strongest luminosity gradient along a projected edge (which can lead to miscorrespondence due to similar strong edges elsewhere), the method considers multiple candidates. It identifies local maxima in the luminosity gradient along search lines perpendicular to projected edges. These candidates are then grouped, and regression lines are fitted to each group. To select the best edge line candidate, it analyzes luminosity gradients in both horizontal and perpendicular directions relative to the candidate lines. This process effectively filters out misleading edges caused by textured surfaces or checkerboard patterns.
  • Hybridization for Robustness: To overcome the limitations of natural feature tracking during rapid camera motion or when features are lost, the system integrates:
  • Magnetic 6DOF Sensor: Provides approximate camera pose information, especially useful when vision tracking fails.
  • Artificial Vision Markers: Placed on the target object, these markers offer a fallback for tracking when natural features are insufficient.
  • The system uses these auxiliary sensors when tracking fails or when camera movement is too fast for reliable natural feature detection.
  • Optimization: The final camera pose (rotation and translation) is refined using the Steepest Descent Method, minimizing the sum of squared projection errors between 3D model points and their corresponding 2D image features.

Datasets

Not discussed in the paper. The experiments appear to be conducted using custom setups with real-world objects (like boxes) and a web camera.

Key Insights and Ideas

  • Intelligent Edge Handling: The most significant insight is the move away from selecting a single "best" edge point. By considering multiple edge candidates and using a multi-stage filtering process based on luminosity gradients, the method dramatically improves robustness against misleading edges, a common problem in vision-based registration.
  • Leveraging Vertices as Anchors: Using vertices, which are geometrically invariant to viewpoint changes, provides stable reference points for initial pose estimation and refinement.
  • Synergistic Hybridization: The paper demonstrates the power of combining different sensing modalities. Natural features provide precision, while magnetic sensors or vision markers offer robustness against tracking loss and rapid motion, creating a more reliable AR system.
  • Model-Based Precision: By utilizing a known 3D model, the system can project features accurately and then search for their real-world counterparts, leading to high registration accuracy.

Key Results

  • Accurate Registration: The method achieves accurate geometric registration, visually demonstrated by the precise superimposition of virtual objects onto real-world scenes in various scenarios.
  • Robustness to Rapid Motion: The hybrid approach, particularly with the inclusion of magnetic sensors or vision markers, allows the system to recover from tracking failures caused by rapid camera movements.
  • Improved Edge Matching: The edge matching process successfully identifies correct edges even in the presence of strong misleading contours (e.g., checkerboard patterns), as shown in Figure 13.
  • Quantitative Error: The paper reports an average projection error of approximately 2.65 pixels in one experiment (Figure 12), indicating good accuracy.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the accuracy of projecting 3D model edges onto true edges in input images needs improvement.
  • While vision markers are used for robustness, the paper notes that reducing constraints on their arrangement would be beneficial.
  • The method relies on having a pre-existing 3D model of the target object.
  • Future Work:
    • Improving the accuracy of edge projection.
  • Reducing the constraints on vision marker placement.
  • Further exploration of hybrid approaches for enhanced AR applications.

Summary updated

BibTeX
@inproceedings{hirose2005vision,
  author = {Ryo Hirose and Hideo Saito},
  title = {{A vision-based AR registration method utilizing edges and vertices of 3D model}},
  booktitle = {ICAT},
  year = {2005},
  doi = {10.1145/1152399.1152433},
}
Figure