View-independent scene acquisition for tele-presence

Jane Mulligan, Kostas Daniilidis

TL;DR

This paper introduces a novel approach to real-time 3D scene acquisition for tele-immersion, enabling users to share a virtual space with remote participants. By combining view-independent stereo techniques with motion prediction, they achieve accurate and robust 3D reconstructions at high frame rates, crucial for a convincing sense of presence.

What Problem Does This Paper Address?

The paper tackles the challenge of creating a truly immersive tele-presence experience. Existing video conferencing lacks depth and 3D interaction, while early tele-immersion systems struggled with real-time, high-quality 3D reconstruction of dynamic scenes. The core problem is acquiring dense, accurate 3D information of remote participants and their environment in real-time, allowing for view-independent rendering and a convincing sense of shared presence, without long processing or network latencies. This is crucial for enabling natural interaction and communication between physically distant individuals.

Methods

The authors propose a view-independent scene acquisition strategy. Instead of rendering based on the viewer's current perspective, they acquire a 3D scene snapshot and transmit it relative to a world coordinate system. This allows for efficient rendering from new viewpoints using simple transformations.

Key techniques include:

  • Bi- and Trinocular Stereo: They employ both binocular and trinocular stereo methods to compute dense depth maps.
  • Sum of Absolute Differences (SAD) and Modified Normalized Cross-Correlation (MNCC): These are used as correlation metrics for stereo matching, with MNCC offering superior depth map quality.
  • Background Subtraction: To speed up processing, they pre-acquire background scenes and use background subtraction to segment out dynamic foreground elements (participants). This reduces the number of pixels that need to be processed for depth calculation.
  • Motion-based Enhancements (Flow per Window): To further optimize real-time performance, they integrate optical flow with disparity calculations. They segment the disparity map into windows with similar depth ranges and use optical flow to predict window movement in subsequent frames. This allows for a reduced disparity search range per window, significantly speeding up the matching process.
  • Parallelized System Architecture: The system is designed with a parallel architecture, distributing processing across multiple servers to handle the computational demands of real-time reconstruction.

Datasets

Not discussed in the paper. The authors describe their system's operation using their own experimental setup and demonstrations, rather than relying on pre-existing public datasets for evaluation.

Key Insights and Ideas

  • View-Independent Acquisition is Key for Tele-Presence: Transmitting a scene description in a world coordinate system, rather than from a specific viewpoint, is crucial for efficient and flexible rendering of remote environments.
  • Balancing Speed and Accuracy: The paper highlights the trade-off between stereo matching algorithms (SAD vs. MNCC) and the need for both speed and high-quality depth data. Trinocular stereo, while computationally more expensive, offers improved depth map quality.
  • Background Subtraction as a Performance Booster: Effectively segmenting static background from dynamic foreground is a critical step for achieving real-time performance in tele-immersion.
  • Integrating Optical Flow with Stereo for Efficiency: The novel approach of using optical flow to predict and constrain disparity search windows is a significant contribution. This "flow per window" method dramatically reduces the computational burden by focusing search efforts on relevant regions and disparity ranges.
  • The "Fooling" of Senses: The ultimate goal is to create a perceptual illusion where users feel physically present with remote participants, a feat that requires not just visual fidelity but also real-time responsiveness.

Key Results

  • Real-time Performance: The system achieves a throughput of almost 2 frames per second (fps) with their seven-camera cluster, with the goal of reaching 10fps.
  • Speedup from Optimization: The window-based correspondence method, leveraging optical flow, reduces computation by approximately half compared to full-image/full-disparity matching.
  • Improved Depth Quality: Trinocular MNCC demonstrated a reduction in noise speckle and refinement in profile detail compared to binocular MNCC.
  • Experimental Validation: The system was demonstrated in a full-scale tele-immersion setup, showing its feasibility for real-world applications.

Limitations and Future Work

  • Outlier Handling: The authors acknowledge that highlights and occlusions are challenges that need to be addressed in future work to further eliminate outliers.
  • Achieving Higher Frame Rates: While 2fps is demonstrated, the goal is to reach 10fps without sacrificing depth quality.
  • System Completeness: The described setup is a partial implementation of the full telecubicle, with participating sites having different configurations.
  • Systematic Experiments: The ultimate goal is to conduct systematic experiments to study the sense of presence and the nuances of communication during collaboration tasks.

Summary updated

Abstract

Tele-immersion is a new medium that enables a user to share a virtual space with remote participants. The user is immersed in a rendered 3D-world that is transmitted from a remote site. To acquire this 3D description we apply bi- and trinocular stereo techniques. The challenge is to compute dense stereo range data at high frame rates, since participants cannot easily communicate if the processing cycle or network latencies are long. Moreover, new views of the received 3D-world must be as accurate as possible. We address both issues of speed and accuracy and we propose a method for combining motion and stereo in order to increase speed and robustness.
BibTeX
@inproceedings{mulligan2000view,
  author = {Jane Mulligan and Kostas Daniilidis},
  title = {{View-independent scene acquisition for tele-presence}},
  booktitle = {ISAR},
  year = {2000},
  doi = {10.1109/isar.2000.880933},
}
Figure