Real time trinocular stereo for tele-immersion

Jane Mulligan, Kostas Daniilidis

TL;DR

This paper introduces a novel real-time trinocular stereo algorithm for tele-immersion, enabling richer 3D interaction between remote users. By combining trinocular vision with motion-based enhancements, they achieve faster and more robust 3D scene reconstruction, crucial for creating a convincing shared virtual experience.

What Problem Does This Paper Address?

Tele-immersion aims to bridge physical distances by creating a shared virtual environment where users can interact as if they were in the same room. A key challenge in achieving this is accurately capturing and rendering the 3D geometry of remote participants and their surroundings in real-time. Traditional videoconferencing lacks depth, and existing 3D reconstruction methods can be too slow or lack the robustness needed for dynamic, interactive scenarios. This paper tackles the problem of acquiring high-quality, real-time 3D visual data for tele-immersion, specifically focusing on reconstructing the dynamic foreground elements (people) in a scene.

Methods

The core of the paper is a new algorithm for real-time trinocular stereo for 3D scene acquisition. Here's a breakdown of their approach:

  • Trinocular Stereo: Instead of just two cameras (binocular stereo), they use three. This provides an additional constraint (the trifocal constraint) that helps to refine correspondences and improve the accuracy and robustness of depth estimation, especially in challenging areas.
  • Background Subtraction: To focus computational resources on the dynamic parts of the scene (people), they first segment the static background. This is done by pre-computing an average background image and then using a difference threshold to identify foreground pixels. This significantly reduces the search space for stereo matching.
  • Correlation Matching: They explore and compare different correlation-based matching techniques, specifically Sum of Absolute Differences (SAD) and Modified Normalized Cross-Correlation (MNCC), for both binocular and trinocular setups.
  • Motion-Based Enhancements: To further speed up the process, they integrate motion estimation. They segment the disparity map into regions with similar depth and then use optical flow to track these regions across frames. This allows them to predict the location of these regions in subsequent frames and significantly reduce the search range for disparities, thereby speeding up the matching process.
  • Rectification and Filtering: Standard image processing techniques like rectification (aligning epipolar lines horizontally) and median filtering are used to simplify matching and reduce noise in the depth maps.

Datasets

Not discussed in the paper. The authors describe their system and experiments using their own setup and data, but no specific public datasets are mentioned for training or evaluation.

Key Insights and Ideas

  • Trinocular vision is a significant step up for robustness: The trifocal constraint provided by a third camera dramatically improves depth map quality by reducing noise and refining details, even though it increases computation.
  • Focusing on the foreground is key for tele-immersion: By intelligently segmenting the static background, computational effort can be concentrated on reconstructing the dynamic elements (people), which are crucial for interactive telepresence.
  • Motion and stereo are powerful allies: Combining motion estimation with stereo matching allows for dynamic adaptation of the search space, leading to substantial speedups without sacrificing too much accuracy. This is a crucial insight for achieving real-time performance.
  • Balancing speed and quality is an ongoing challenge: The paper highlights the trade-offs between different stereo matching algorithms (SAD vs. MNCC) and the overall system's performance, emphasizing that achieving both high speed and high quality requires careful algorithm selection and integration.

Key Results

  • Improved Depth Map Quality: Trinocular MNCC produced significantly less noise and more refined depth maps compared to binocular MNCC, as visually demonstrated in their figures.
  • Real-time Performance Achieved: While specific frame rates are not explicitly stated as a final system metric, the timings for individual components (Table 1) show that their methods, when optimized, can operate at speeds suitable for real-time applications (e.g., SAD-based methods achieving 3.8 fps for a 320x240 image with 60 disparities).
  • Motion-based enhancements reduce computation: The proposed motion-based approach aims to reduce the complexity of stereo correspondence by roughly half, depending on the number of frames used for optical flow estimation.
  • Experimental validation: They demonstrated that their window-based reconstructions, even after several frames of propagation via estimated optical flow, compared favorably to full-image/full-disparity reconstructions, with mean differences in computed disparities less than 1 pixel.

Limitations and Future Work

  • Limitations:
    • The current system is one-way (acquisition site only).
  • The static background is transmitted once and assumed to remain constant for the session.
  • The avatar-based tele-collaboration is still at the level of cartoon-like representations, suggesting a need for more sophisticated 3D models.
  • The motion-based enhancement relies on the assumption of image brightness constancy, which might not hold in all scenarios.
  • Future Work:
    • Merging their methods with 3D graphical descriptions of remote participants (avatars) to create more detailed and interactive representations.
  • Developing more sophisticated methods for extracting detailed human models from the raw depth points.
  • Extending the system to include other modalities like audio and haptics for a more complete tele-immersion experience.
  • Further optimization of the motion-based enhancements for even greater speed and robustness.

Summary updated

Abstract

Tele-immersion is a technology that augments your space with real-time 3D projections of remote spaces thus facilitating the interaction of people from different places in virtually the same environment. Tele-immersion combines 3D scene recovery from computer vision, and rendering and interaction from computer graphics. We describe the real-time 3D scene acquisition using a new algorithm for trinocular stereo. We extend this method in time by combining motion and stereo in order to increase speed and robustness.
BibTeX
@inproceedings{mulligan2001real,
  author = {Jane Mulligan and Kostas Daniilidis},
  title = {{Real time trinocular stereo for tele-immersion}},
  booktitle = {ICIP},
  year = {2001},
  doi = {10.1109/icip.2001.958284},
}
Figure