Abstract
Achieving realistic Augmented Reality (AR) requires seamless integration of virtual objects into physical environments, particularly through accurate occlusion. Traditional methods rely on regressing full 3D depth maps, which often suffer from noise and flickering at object boundaries. This paper presents a novel "implicit depth" model that treats occlusion as a binary classification problem conditioned on a query depth. By predicting whether a virtual object is in front of or behind the real world at a pixel level, the method achieves sharper edges and significantly improved temporal stability.
Introduction: The Challenge of AR Occlusion
For an AR experience to be immersive, virtual objects must interact naturally with physical geometry. Occlusion—the phenomenon where a real-world object hides a virtual one—is fundamental to this interaction. The standard industry approach involves estimating a per-pixel depth map of the environment and comparing it to the depth of virtual assets.
However, depth regression is a difficult task for neural networks. Small inaccuracies in distance estimation, particularly around thin structures like chair legs or foliage, result in noisy occlusion masks. In video sequences, these errors manifest as distracting flickering, breaking the illusion of presence.
Methodology: Occlusion as Classification
The researchers propose a shift from continuous depth regression to binary classification. Instead of asking "How far away is this pixel?", the model asks: "At this specific pixel and virtual depth, is the real world in front?"
Implicit Depth Model
The architecture consists of a backbone network that extracts features from a color image and a lightweight Multi-Layer Perceptron (MLP). The MLP takes these features along with the known depth of the virtual object to predict a final compositing mask. This classification task is inherently more robust than absolute distance regression.
Temporal Stability and Soft Blending
To address flickering, the model incorporates temporal feedback. Previous predictions are warped based on camera movement and fed back into the network, encouraging consistency across frames. Furthermore, the model utilizes a sigmoid layer to output probabilities, allowing for soft blending at edges. This produces a more natural visual transition than the hard cuts typical of standard depth testing.
Evaluation and Key Results
The model was trained on a combination of real-world (ScanNetv2) and synthetic (Hypersim) datasets to balance realistic complexity with high-fidelity edge supervision.
The proposed method demonstrates significant improvements over existing baselines:
- Accuracy: On the ScanNetv2 benchmark, the model outperformed traditional regression techniques in both occlusion quality (Intersection-over-Union) and absolute depth accuracy.
- Stability: The temporal smoothing mechanism reduced flickering by approximately 30% compared to standard methods.
- Fine-Grained Detail: Qualitatively, the model handles challenging geometries—such as the gaps in a wooden bench or thin plant stems—better than the Lidar sensors found in high-end mobile devices.
Discussion and Future Work
A unique property of this implicit model is its ability to generate standard depth maps through binary search along camera rays. This approach achieved state-of-the-art results on ScanNetv2, suggesting that classification-based depth estimation is a powerful alternative to direct regression.
Despite these gains, limitations remain. The model currently struggles with transparent surfaces due to limitations in common training datasets. Future work may explore end-to-end pixel generation to handle complex lighting and shadows, as well as dedicated training for dynamic scenes containing moving real-world objects.
Summary updated