Abstract: For augmented reality (AR), it is important that virtual assets appear to ‘sit among’ real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer's camera moves. Unfortunately, small mistakes in the estimated scene depth can ruin the downstream occlusion mask, and thereby the AR illusion. Especially in real-time settings, depths inferred near boundaries or across time can be inconsistent. In this paper, we challenge the need for depth-regression as an intermediate step. We instead propose an implicit model for depth and use that to predict the occlusion mask directly. The inputs to our network are one or more color images, plus the known depths of any virtual geometry. We show how our occlusion predictions are more accurate and more temporally stable than predictions derived from traditional depth-estimation models. We obtain state-of-the-art occlusion results on the challenging ScanNetv2 dataset and superior qualitative results on real scenes.
Introduction
In Augmented Reality (AR), the "illusion of presence" is fragile. For a virtual character to look like it is truly part of a physical environment, it must interact realistically with its surroundings. Most critically, it must be occluded by real-world objects like furniture or walls.
Traditionally, this is achieved by estimating the depth of every pixel in the real scene and comparing it to the virtual object's depth. However, this "depth-regression bottleneck" often leads to visual artifacts such as flickering edges and virtual objects bleeding through solid surfaces. This research, a collaboration between Niantic, the University of Edinburgh, and UCL, proposes a shift in perspective: treating depth as an implicit query to directly predict occlusion masks.
Methodology: Implicit Depth for Occlusion
Instead of regressing a full depth map, the proposed model directly predicts occlusion masks. The core idea is to ask the network a binary question: "Is the real world in front of or behind a virtual object at a specific depth $D$?"
Architecture and Temporal Stability
The system utilizes an image encoder backbone (based on SimpleRecon) to extract features from the current and preceding frames. A lightweight Multi-Layer Perceptron (MLP) then processes these features along with the known depth of the virtual geometry ($D_{virtual}$) and a warped occlusion prediction from the previous frame.
By conditioning the network on $D_{virtual}$, the task becomes a binary classification (occluded vs. visible) rather than continuous regression. This classification approach is mathematically simpler for the network, leading to sharper, more confident boundaries. To ensure temporal stability and reduce flickering, camera motion data is used to warp previous masks into the current perspective, providing the network with a consistent "memory" of the scene.
Evaluation and Results
The model was rigorously tested on datasets including ScanNetv2, Hypersim, and 7Scenes, as well as casual footage from mobile devices.
Key Findings
- Superior Edge Fidelity: The model significantly outperforms existing depth-regression methods, particularly around thin structures like chair legs or foliage where traditional methods often fail.
- State-of-the-Art Performance: On the ScanNetv2 benchmark, the method achieved top results for both occlusion Intersection over Union (IoU) and traditional depth metrics.
- Reduced Flickering: Temporal stability tests showed a nearly 30% reduction in flickering compared to standard regression baselines.
- Depth as a Side Effect: While optimized for occlusion, the model can generate high-quality depth maps via a binary search over the implicit depth function, achieving state-of-the-art depth estimation results.
Conclusion and Future Work
This implicit depth approach offers a robust solution for realistic AR occlusions. While the model handles static scenes with high precision, future work may address challenges such as transparent objects (e.g., glass) and highly dynamic environments. Integrating this model with lighting and shadow estimation could eventually lead to a fully end-to-end AR compositing pipeline.
Citation: Watson, J., Sayed, M., Qureshi, Z., Brostow, G. J., Vicente, S., Mac Aodha, O., & Firman, M. (2023). Virtual Occlusions Through Implicit Depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Summary updated