Heightfields for Efficient Scene Reconstruction for AR

Jamie Watson, Sara Vicente, Oisin Mac Aodha, Clément Godard, Gabriel J. Brostow, Michael FirmanWACV 2023 · OpenReview

Abstract

3D scene reconstruction from a sequence of posed RGB images is a cornerstone task for computer vision and augmented reality (AR). While depth-based fusion is the foundation of most real-time approaches for 3D reconstruction, recent learning-based methods that operate directly on RGB images can achieve higher quality reconstructions, but at the cost of increased runtime and memory requirements, making them unsuitable for AR applications. We propose an efficient learning-based method that refines the 3D reconstruction obtained by a traditional fusion approach. By leveraging a top-down heightfield representation, our method remains real-time while approaching the quality of other learning-based methods. Despite being a simplification, our heightfield is perfectly appropriate for robotic path planning or augmented reality character placement. We outline several innovations that push the performance beyond existing top-down prediction baselines, and we present an evaluation framework on the challenging ScanNetV2 dataset, targeting AR tasks.

Introduction

For Augmented Reality (AR) to feel immersive, virtual characters must interact convincingly with the physical world—standing on tables, walking behind sofas, or navigating complex floor plans. This requires accurate 3D reconstruction of the environment. Currently, researchers face a trade-off: traditional depth fusion methods are fast and efficient but often produce noisy, incomplete reconstructions. Conversely, state-of-the-art deep learning models like Atlas or NeuralRecon produce high-quality, complete meshes but rely on computationally expensive 3D convolutions. These memory-intensive operations are ill-suited for the limited hardware of mobile devices.

HeightRecon identifies a middle ground. In most indoor environments, geometry varies significantly across the floor (horizontally) but remains relatively simple vertically. By adopting a 2.5D "heightfield" representation—a top-down 2D grid of heights—the system can achieve 3D-quality results at 2D processing speeds.

Methods

HeightRecon operates through a four-stage pipeline designed for efficiency and accuracy:

  1. Per-image Estimation: For every incoming RGB frame, the system predicts a depth map and extracts deep 2D features.
  2. Raw TSDF Fusion: These depth maps are fused into a "raw" Truncated Signed Distance Function (TSDF) volume, providing a fast baseline for 3D data aggregation.
  3. Top-down Feature Aggregation: Instead of processing a full 3D volume, the system creates a 2D top-down feature map. By raycasting the raw TSDF from above to find the surface, image features are projected onto specific 2D grid locations, avoiding the "smearing" common in other top-down methods.
  4. Learned Blending & Regression: A 2D CNN processes the feature map to predict a refined heightfield and a "blend map." This allows the system to prioritize sharp, raw sensor data where available and rely on the network's "hallucination" to fill in holes.
Overview of the HeightRecon pipeline, qualitative comparisons with Atlas and NeuralRecon, and performance trade-offs.

Key Results

The system was evaluated on the ScanNetV2 dataset and a custom AR-style protocol that measures Rendering IoU (occlusion accuracy) and Placement IoU (accuracy of object placement on surfaces).

  • Efficiency: HeightRecon runs at approximately 33ms per update (~30 FPS) on a desktop GPU. This is significantly faster than non-real-time 3D alternatives like Atlas, which requires 382ms per update. Peak memory usage is reduced by up to 10x compared to 3D models.
  • Accuracy: Despite the 2.5D simplification, HeightRecon outperforms all other real-time baselines. It achieves an F1 score of 0.680, remaining competitive with heavy 3D models like Atlas (0.750).
  • AR Performance: In placement tasks, HeightRecon achieved a Placement IoU of 0.727, nearly matching Atlas (0.768) and significantly outperforming raw depth fusion (0.628).
Quantitative comparison showing F1 score versus update time. HeightRecon (Ours) occupies the high-performance, real-time quadrant.

Discussion and Limitations

The core innovation of HeightRecon is the use of a 2.5D prior. For tasks like character placement and pathfinding, a full 3D voxel grid is often unnecessary detail. However, the heightfield representation does have limitations: it cannot represent the space underneath objects (e.g., the area under a table) or overhanging structures like ceiling fans.

Future work may explore "layered" representations—using multiple height values per grid cell—to handle complex furniture more gracefully without sacrificing the efficiency of the 2D grid. Currently, the model provides a robust, real-time solution for high-quality AR experiences on mobile hardware.

Summary updated

Abstract

3D scene reconstruction from a sequence of posed RGB images is a cornerstone task for computer vision and augmented reality (AR). While depth-based fusion is the foundation of most real-time approaches for 3D reconstruction, recent learning based methods that operate directly on RGB images can achieve higher quality reconstructions, but at the cost of increased runtime and memory requirements, making them unsuitable for AR applications. We propose an efficient learning-based method that refines the 3D reconstruction obtained by a traditional fusion approach. By leveraging a top-down heightfield representation, our method remains real-time while approaching the quality of other learning-based methods. Despite being a simplification, our heightfield is perfectly appropriate for robotic path planning or augmented reality character placement. We outline several innovations that push the performance beyond existing top-down prediction baselines, and we present an evaluation framework on the challenging ScanNetV2 dataset, targeting AR tasks.
BibTeX
@inproceedings{watson2023heightfields,
  author = {Jamie Watson and Sara Vicente and Oisin Mac Aodha and Clément Godard and Gabriel J. Brostow and Michael Firman},
  title = {{Heightfields for Efficient Scene Reconstruction for AR}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=HVAPHUhRfZ},
}
Figure