Virtual Occlusions Through Implicit Depth

Jamie Watson, Mohamed Sayed, Zawar Qureshi, Gabriel J. Brostow, Sara Vicente, Oisin Mac Aodha, Michael Firman

Abstract

Achieving realistic Augmented Reality (AR) requires seamless integration of virtual objects into physical environments, particularly through accurate occlusion. Traditional methods rely on regressing full 3D depth maps, which often suffer from noise and flickering at object boundaries. This paper presents a novel "implicit depth" model that treats occlusion as a binary classification problem conditioned on a query depth. By predicting whether a virtual object is in front of or behind the real world at a pixel level, the method achieves sharper edges and significantly improved temporal stability.

Introduction: The Challenge of AR Occlusion

For an AR experience to be immersive, virtual objects must interact naturally with physical geometry. Occlusion—the phenomenon where a real-world object hides a virtual one—is fundamental to this interaction. The standard industry approach involves estimating a per-pixel depth map of the environment and comparing it to the depth of virtual assets.

However, depth regression is a difficult task for neural networks. Small inaccuracies in distance estimation, particularly around thin structures like chair legs or foliage, result in noisy occlusion masks. In video sequences, these errors manifest as distracting flickering, breaking the illusion of presence.

Methodology: Occlusion as Classification

The researchers propose a shift from continuous depth regression to binary classification. Instead of asking "How far away is this pixel?", the model asks: "At this specific pixel and virtual depth, is the real world in front?"

Implicit Depth Model

The architecture consists of a backbone network that extracts features from a color image and a lightweight Multi-Layer Perceptron (MLP). The MLP takes these features along with the known depth of the virtual object to predict a final compositing mask. This classification task is inherently more robust than absolute distance regression.

Temporal Stability and Soft Blending

To address flickering, the model incorporates temporal feedback. Previous predictions are warped based on camera movement and fed back into the network, encouraging consistency across frames. Furthermore, the model utilizes a sigmoid layer to output probabilities, allowing for soft blending at edges. This produces a more natural visual transition than the hard cuts typical of standard depth testing.

Evaluation and Key Results

The model was trained on a combination of real-world (ScanNetv2) and synthetic (Hypersim) datasets to balance realistic complexity with high-fidelity edge supervision.

Comparison of occlusion quality between ARKit Lidar, depth regression, and the proposed implicit depth method.

The proposed method demonstrates significant improvements over existing baselines:

  • Accuracy: On the ScanNetv2 benchmark, the model outperformed traditional regression techniques in both occlusion quality (Intersection-over-Union) and absolute depth accuracy.
  • Stability: The temporal smoothing mechanism reduced flickering by approximately 30% compared to standard methods.
  • Fine-Grained Detail: Qualitatively, the model handles challenging geometries—such as the gaps in a wooden bench or thin plant stems—better than the Lidar sensors found in high-end mobile devices.

Discussion and Future Work

A unique property of this implicit model is its ability to generate standard depth maps through binary search along camera rays. This approach achieved state-of-the-art results on ScanNetv2, suggesting that classification-based depth estimation is a powerful alternative to direct regression.

Despite these gains, limitations remain. The model currently struggles with transparent surfaces due to limitations in common training datasets. Future work may explore end-to-end pixel generation to handle complex lighting and shadows, as well as dedicated training for dynamic scenes containing moving real-world objects.

Summary updated

Abstract

For augmented reality (AR), it is important that virtual assets appear to `sit among' real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer's camera moves. Unfortunately, small mistakes in the estimated scene depth can ruin the downstream occlusion mask, and thereby the AR illusion. Especially in real-time settings, depths inferred near boundaries or across time can be inconsistent. In this paper, we challenge the need for depth-regression as an intermediate step. We instead propose an implicit model for depth and use that to predict the occlusion mask directly. The inputs to our network are one or more color images, plus the known depths of any virtual geometry. We show how our occlusion predictions are more accurate and more temporally stable than predictions derived from traditional depth-estimation models. We obtain state-of-the-art occlusion results on the challenging ScanNetv2 dataset and superior qualitative results on real scenes.
BibTeX
@inproceedings{watson2023virtual,
  author = {Jamie Watson and Mohamed Sayed and Zawar Qureshi and Gabriel J. Brostow and Sara Vicente and Oisin Mac Aodha and Michael Firman},
  title = {{Virtual Occlusions Through Implicit Depth}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=W9vi975HuM},
}
Figure