Mixed Reality Depth Contour Occlusion Using Binocular Similarity Matching and Three-dimensional Contour Optimisation

Naye Ji, Fan Zhang, Haoxiang Zhang, Youbing Zhao, Dingguo Yu

Abstract

Mixed reality (MR) applications often require virtual objects to be realistically occluded by real-world objects. However, achieving high-quality occlusion in real-time remains a challenge for many systems. This paper proposes a novel Depth Contour Occlusion (DCO) algorithm designed for binocular stereoscopic vision. The method integrates a depth contour map—extracted using Digital Image Stabilisation (DIS) optical flow—with a sparse depth map generated via a two-stage adaptive AD-census stereo matching algorithm. To produce a precise dense depth map suitable for high-quality occlusion, we employ a quadratic optimization model with three specific constraints. The entire pipeline is GPU-accelerated, enabling near real-time performance. Experimental results across various occlusion scenarios demonstrate that the DCO algorithm significantly improves both the visual quality and the efficiency of real-virtual object interactions in MR environments.

Introduction

In Mixed Reality, "occlusion" is the critical ability of a real-world object to hide a virtual object when positioned in front of it. Maintaining geometric consistency is essential for a convincing user experience; the system must accurately determine the distance of every pixel in the scene.

Current solutions often fall into one of three categories, each with its own drawbacks:

  1. Professional Hardware: Devices like the Microsoft HoloLens utilize dedicated depth sensors (LiDAR or RGB-D). While effective, these sensors increase the cost, weight, and power consumption of the device.
  2. Software-only Monocular Approaches: Frameworks like ARKit or ARCore use machine learning on single-camera feeds. These can struggle with edge-fitting and accuracy in complex or dynamic environments.
  3. Full 3D Reconstruction: Building a complete 3D model of an environment is computationally intensive and often too slow for interactive, low-latency MR applications.

This research identifies that for realistic occlusion, contour accuracy—the sharpness and alignment of edges—is more vital than absolute depth precision. Misaligned or blurry edges immediately break the illusion of reality.

Methodology

The proposed DCO algorithm utilizes a multi-stage pipeline optimized for GPU execution:

1. Binocular Stereo Matching

The system employs a two-stage adaptive "AD-census" algorithm to calculate similarity between the left and right camera views. This process generates a sparse depth map. To maintain high performance, the images are processed at a quarter of their original resolution.

2. Depth Contour Extraction

A key innovation of this work is the use of Digital Image Stabilisation (DIS) optical flow to extract contours. By analyzing a three-frame window (past, current, and future), the system can distinguish moving 3D boundaries from static surface textures (like patterns on a wall). This ensures that the occlusion edges correspond to actual physical object boundaries.

The overall architecture of the DCO algorithm, illustrating the parallel processing of stereo matching and contour extraction.

3. Quadratic Optimisation

The sparse depth data and extracted contours are combined in a mathematical model governed by three constraints:

  • Data Constraint: Keeps the final map consistent with the initial sparse depth readings.
  • Smoothness Constraint: Aligns depth discontinuities with the extracted object contours to ensure sharp, clean edges.
  • Stability Constraint: Incorporates temporal information from previous frames to minimize flickering and jitter.

4. GPU Acceleration

The entire pipeline, including the resolution of Poisson-based linear equations during optimization, is implemented using NVIDIA CUDA. This allows the system to achieve the speeds necessary for interactive Mixed Reality.

Experimental Results

The algorithm was evaluated using the standard Middlebury 2014 benchmark and a custom dataset captured with an INDEMIND binocular camera (1280x720 resolution).

Visual Quality and Robustness

Qualitative analysis shows that the DCO method produces significantly more accurate occlusion edges compared to traditional stereo matching. It effectively handles complex scenarios, such as "enclosed" occlusions (objects inside containers) and "grasping" motions, where sparse methods typically fail. The optimization step is crucial for "filling in" the sparse map while maintaining razor-sharp boundaries guided by the extracted contours.

Dense depth map results for various scenes: (a) Stairs, (b) Nkuba, and (c) Class, demonstrating the algorithm's ability to handle different indoor environments.

Performance

On a system with an NVIDIA GTX 1080Ti, the average processing time per frame was 82.43 ms (~12.13 fps). While not yet reaching the 30-60 fps standard for high-end MR, it represents a significant advancement for software-based depth handling on standard binocular hardware.

Conclusion and Future Work

The DCO algorithm provides a robust framework for real-time depth handling in MR without requiring specialized depth sensors. By prioritizing contour accuracy and leveraging GPU acceleration, it achieves high-quality occlusion effects. Future work will focus on further optimizing the GPU implementation to reach 30+ fps, integrating the system with SLAM for better spatial awareness, and exploring deep learning models for even more precise contour extraction.

Summary updated

Abstract

Mixed reality applications often require virtual objects that are partly occluded by real objects. However, previous research and commercial products have limitations in terms of performance and efficiency. To address these challenges, we propose a novel depth contour occlusion (DCO) algorithm. The proposed method is based on the sensitivity of contour occlusion and a binocular stereoscopic vision device. In this method, a depth contour map is combined with a sparse depth map obtained from a two-stage adaptive filter area stereo matching algorithm and the depth contour information of the objects extracted by a digital image stabilisation optical flow method. We also propose a quadratic optimisation model with three constraints to generate an accurate dense map of the depth contour for high-quality real-virtual occlusion. The whole process is accelerated by GPU. To evaluate the effectiveness of the algorithm, we demonstrate a time con-sumption statistical analysis for each stage of the DCO algorithm execution. To verify the relia-bility of the real-virtual occlusion effect, we conduct an experimental analysis on single-sided, enclosed, and complex occlusions; subsequently, we compare it with the occlusion method without quadratic optimisation. With our GPU implementation for real-time DCO, the evaluation indicates that applying the presented DCO algorithm can enhance the real-time performance and the visual quality of real-virtual occlusion.
BibTeX
@misc{ji2022mixed,
  author = {Naye Ji and Fan Zhang and Haoxiang Zhang and Youbing Zhao and Dingguo Yu},
  title = {{Mixed Reality Depth Contour Occlusion Using Binocular Similarity Matching and Three-dimensional Contour Optimisation}},
  year = {2022},
  eprint = {2203.02300},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.02300},
}
Figure