Abstract
Mixed reality (MR) applications often require virtual objects to be realistically occluded by real-world objects. However, achieving high-quality occlusion in real-time remains a challenge for many systems. This paper proposes a novel Depth Contour Occlusion (DCO) algorithm designed for binocular stereoscopic vision. The method integrates a depth contour map—extracted using Digital Image Stabilisation (DIS) optical flow—with a sparse depth map generated via a two-stage adaptive AD-census stereo matching algorithm. To produce a precise dense depth map suitable for high-quality occlusion, we employ a quadratic optimization model with three specific constraints. The entire pipeline is GPU-accelerated, enabling near real-time performance. Experimental results across various occlusion scenarios demonstrate that the DCO algorithm significantly improves both the visual quality and the efficiency of real-virtual object interactions in MR environments.
Introduction
In Mixed Reality, "occlusion" is the critical ability of a real-world object to hide a virtual object when positioned in front of it. Maintaining geometric consistency is essential for a convincing user experience; the system must accurately determine the distance of every pixel in the scene.
Current solutions often fall into one of three categories, each with its own drawbacks:
- Professional Hardware: Devices like the Microsoft HoloLens utilize dedicated depth sensors (LiDAR or RGB-D). While effective, these sensors increase the cost, weight, and power consumption of the device.
- Software-only Monocular Approaches: Frameworks like ARKit or ARCore use machine learning on single-camera feeds. These can struggle with edge-fitting and accuracy in complex or dynamic environments.
- Full 3D Reconstruction: Building a complete 3D model of an environment is computationally intensive and often too slow for interactive, low-latency MR applications.
This research identifies that for realistic occlusion, contour accuracy—the sharpness and alignment of edges—is more vital than absolute depth precision. Misaligned or blurry edges immediately break the illusion of reality.
Methodology
The proposed DCO algorithm utilizes a multi-stage pipeline optimized for GPU execution:
1. Binocular Stereo Matching
The system employs a two-stage adaptive "AD-census" algorithm to calculate similarity between the left and right camera views. This process generates a sparse depth map. To maintain high performance, the images are processed at a quarter of their original resolution.
2. Depth Contour Extraction
A key innovation of this work is the use of Digital Image Stabilisation (DIS) optical flow to extract contours. By analyzing a three-frame window (past, current, and future), the system can distinguish moving 3D boundaries from static surface textures (like patterns on a wall). This ensures that the occlusion edges correspond to actual physical object boundaries.
3. Quadratic Optimisation
The sparse depth data and extracted contours are combined in a mathematical model governed by three constraints:
- Data Constraint: Keeps the final map consistent with the initial sparse depth readings.
- Smoothness Constraint: Aligns depth discontinuities with the extracted object contours to ensure sharp, clean edges.
- Stability Constraint: Incorporates temporal information from previous frames to minimize flickering and jitter.
4. GPU Acceleration
The entire pipeline, including the resolution of Poisson-based linear equations during optimization, is implemented using NVIDIA CUDA. This allows the system to achieve the speeds necessary for interactive Mixed Reality.
Experimental Results
The algorithm was evaluated using the standard Middlebury 2014 benchmark and a custom dataset captured with an INDEMIND binocular camera (1280x720 resolution).
Visual Quality and Robustness
Qualitative analysis shows that the DCO method produces significantly more accurate occlusion edges compared to traditional stereo matching. It effectively handles complex scenarios, such as "enclosed" occlusions (objects inside containers) and "grasping" motions, where sparse methods typically fail. The optimization step is crucial for "filling in" the sparse map while maintaining razor-sharp boundaries guided by the extracted contours.
Performance
On a system with an NVIDIA GTX 1080Ti, the average processing time per frame was 82.43 ms (~12.13 fps). While not yet reaching the 30-60 fps standard for high-end MR, it represents a significant advancement for software-based depth handling on standard binocular hardware.
Conclusion and Future Work
The DCO algorithm provides a robust framework for real-time depth handling in MR without requiring specialized depth sensors. By prioritizing contour accuracy and leveraging GPU acceleration, it achieves high-quality occlusion effects. Future work will focus on further optimizing the GPU implementation to reach 30+ fps, integrating the system with SLAM for better spatial awareness, and exploring deep learning models for even more precise contour extraction.
Summary updated