Markerless Vision-Based Augmented Reality for Enhanced Project Visualization

Frédéric Bosché, David Tingdahl, Ludovico Carozza, Luc Van GoolProceedings of the ... ISARC 2012 · DOI

TL;DR

This paper introduces a novel markerless augmented reality (AR) system for the Architectural Engineering and Construction (AEC) industry. It enables the visualization of 3D project data overlaid onto the real world using only a standard camera, overcoming limitations of existing VR and AR solutions by accurately handling occlusions and providing a more integrated experience.

What Problem Does This Paper Address?

The construction industry generates vast amounts of complex data, often managed through Building Information Modeling (BIM). Visualizing this 3D information in the context of the actual construction site is crucial for design, construction, and management. Existing Virtual Reality (VR) solutions are largely confined to digital worlds, making them less useful during the physical construction and operation phases. While Augmented Reality (AR) offers a bridge between virtual and real, current AR systems often struggle with accurate positioning without external markers or sensors, and they frequently fail to account for occlusions (when real-world objects hide virtual ones), leading to unrealistic visualizations. This paper aims to address these limitations by developing an AR system that is solely image-based, markerless, and capable of handling occlusions.

Methods

The proposed system operates in two stages:

1. Offline Mapping Stage: * Scene Learning: A 3D map of the environment is constructed from a set of training images. This involves: * Extracting Speeded Up Robust Features (SURF) from the images. * Using Structure-from-Motion (SfM) techniques with SURF features to recover the 3D structure of the scene. * Employing Euclidean Bundle Adjustment to register 2D SURF descriptors into a 3D reference frame, creating a map of 3D-referenced SURF features. The ARC3D framework is used for this process. * Dense Mesh Reconstruction: Crucially, the system also generates a dense 3D textured mesh of the scene using Poisson mesh reconstruction. This mesh is vital for handling occlusions. * Scene Augmentation: Virtual 3D objects are then registered with this 3D map.

2. Online Image Stream Processing: * Image Positioning: For each incoming target image (e.g., from a video stream), the system estimates the camera's pose (position and orientation). This is achieved by: * Extracting SURF features from the target image. * Matching these features against the pre-computed 3D SURF features in the map. * Using a robust 3-point algorithm with RANSAC (Random Sample Consensus) to estimate the initial camera pose. * A Guided Refinement process further improves pose accuracy. * A Levenberg-Marquardt non-linear optimization algorithm is used for the final pose estimation. * Image Augmentation: Once the camera pose is determined, the virtual objects are rendered and overlaid onto the target image. The dense 3D mesh is used to correctly compute static occlusions, ensuring that virtual objects are hidden by real-world objects as they should be.

To optimize performance for real-time applications, several techniques are employed:

  • K-d Tree: The SURF descriptors in the scene database are organized into a k-d tree to expedite matching.
  • Feature Strength Filtering: SURF features are assigned a global strength value based on their repeatability across multiple images, allowing for prioritization of more reliable features.
  • Multiple Matching: The system allows for one-to-many matches between target image features and database features to improve robustness.
  • Mode-Specific Strategies: The online processing employs three modes:
  • Initialization: Used when no prior pose information is available, employing a pruned octree for efficient scene coverage.
  • Tracking: Utilizes an Extended Kalman Filter (EKF) to predict the camera pose and culls features within the predicted view.
  • Resetting: Used when tracking fails, it re-initializes pose estimation using a sphere around the last known good pose.

Datasets

The paper describes experiments conducted on two real-world scenes:

  • Housing Estate in Edinburgh: A modern housing estate with apartment buildings. The authors acquired 22 training images and 120 target images (simulating a video sequence). Datasets were also downsampled to 640x480 resolution for scale comparison.
  • Seoul Imperial Palace: A historical site. The experiment used 80 training images and 90 target images.

No new benchmarks were introduced; existing real-world scenes were used for evaluation.

Key Insights and Ideas

  • Markerless AR is Achievable and Practical for AEC: The core contribution is demonstrating a robust, markerless AR system for AEC that doesn't rely on external sensors, making it more practical for field deployment.
  • Dense Mesh for Occlusion Handling is Crucial: The integration of dense 3D mesh reconstruction for accurate occlusion calculation is a significant advancement over previous markerless AR approaches. This is highlighted as a key advantage for realistic visualization.
  • SURF Features Offer a Good Balance: The choice of SURF features for both mapping and tracking provides a good balance between robustness to scale changes and computational efficiency, enabling the system to work with varying image resolutions.
  • Multi-Stage Pose Estimation is Effective: The combination of initial RANSAC-based estimation followed by guided refinement and non-linear optimization leads to accurate and stable camera pose estimation.
  • Optimization Strategies are Necessary for Real-Time: The paper clearly shows that without optimizations like k-d trees and feature filtering, real-time performance would be unattainable.

Key Results

  • Accurate and Stable Positioning: Experiments demonstrated that the system achieves accurate and stable camera pose estimations, effectively handling tracking failures and recovering from them.
  • Successful Augmentation with Occlusions: The system successfully augmented images with virtual objects, correctly accounting for static occlusions from the reconstructed scene mesh.
  • Performance with Varying Resolutions: The system performs best when training and target images have similar resolutions. Smaller image sizes (640x480) lead to faster processing times with comparable pose estimation quality.
  • Processing Speed: While described as "near real-time," the paper acknowledges that the current implementation's processing speed (up to 3fps) is not yet sufficient for true real-time AR applications, especially with larger datasets. The use of Guided Refinement increased processing time by approximately 40%.
  • Impact of MM and GR: Multiple Matching (MM) and Guided Refinement (GR) generally improved pose calculation success rates, though the improvement was not always critical in the tested scenarios.

Limitations and Future Work

Limitations: * Processing Speed: The system's current speed (up to 3fps) is not fast enough for real-time AR applications. * Textured Scenes: The system relies on adequately textured scenes for feature detection and matching. It would likely fail in environments with little visual structure (e.g., greenfield projects). * Dense Mesh Quality: In some cases, the dense mesh reconstruction (Poisson) did not achieve sufficient quality, leading to artifacts and poor occlusion calculations (as seen in Experiment 2). * Camera Tracking vs. Feature Tracking: The system tracks the camera rather than image features directly, which might be less efficient than combined feature-based tracking.

Future Work: * GPU Acceleration: Transferring some data processing to the GPU is suggested to improve speed. * Hybrid Systems: Investigating hybrid pose estimation modules that incorporate GPS or inertial systems for initialization and resetting could enhance robustness. * Planar Structure Techniques: For urban AR, exploring positioning techniques based on planar structures could be beneficial. * Feature Visibility Criteria: Further culling of database features using visibility criteria could improve efficiency. * Larger Scenes: Testing the system with larger reconstructed scenes is recommended. * Improved 3D Reconstructions: Utilizing more accurate 3D urban reconstructions (e.g., GIS data) instead of solely relying on SfM for dense mesh generation could improve occlusion handling.

Summary updated

Abstract

Markerless Vision-Based Augmented Reality for Enhanced Project Visualization Frédéric Bosché, David Tingdahl, Ludovico Carozza, Luc Van Gool Abstract: Purpose This work aims to develop a system that enables improved visualization of Architectural Engineering and Con-struction (AEC) 3D-information for application in design, construction, and management of the built environment. Meth-od A novel Augmented Reality (AR) system is presented that uses a single standard digital camera and which, contrary to other investigated approaches, does not rely on any markers inserted in the scene, nor on any positioning and inertial technologies. The system is solely image-based and consists of two stages. In a first offline stage, a 3D-map of the sce-ne is automatically constructed from a set of digital images, and the augmenting information (e.g. the 3D-model of the building asset) is subsequently registered with this map. The 3D-map reconstruction employs structure-from-motion techniques with SURF features (the resulting map consisting of a set 3D-referenced SURF-features) followed by a Pois-son mesh reconstruction procedure. The next step consists of online operations. The positions of target digital images (e.g. from video stream or head-mounted camera) are automatically calculated, using a robust SURF feature matching procedure that is optimized for three different situations (initialization, tracking, and resetting) implementing octrees for efficient 3D-pruning, and kd-trees for efficient feature matching. Once each input image is positioned within the map, the view is augmented. A notable feature of dense mesh scene reconstruction conducted in the present work is that it ena-bles static occlusions of the scene on the augmenting data to be taken into account. Results & Discussion Several experiments validate the pr posed system and demonstrate its overall performance: a near real-time processing speed, very accurate and stable positioning. The limitations of the current system are also discussed including: the currently limited processing speed and the need for adequately textured scenes. Keywords: realities or application systems, augmented reality, image-based, markerless DOI: https://doi.org/10.22260/ISARC2012/0005 Download fulltext Download BibTex Download Endnote (RIS) TeX Import to Mendeley
BibTeX
@inproceedings{bosche2012markerless,
  author = {Frédéric Bosché and David Tingdahl and Ludovico Carozza and Luc Van Gool},
  title = {{Markerless Vision-Based Augmented Reality for Enhanced Project Visualization}},
  booktitle = {Proceedings of the ... ISARC},
  year = {2012},
  doi = {10.22260/isarc2012/0005},
}
Figure