Project starline: a high-fidelity telepresence system

Jason Lawrence, Dan B. Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G. Desloge, Tommy Fortes, Eric M. Gomez, Sascha Häberling, Hugues Hoppe, Andy Huibers, Claude Knaus, Brian Kuschak, Ricardo Martin-Brualla, Harris Nover, Andrew Ian Russell, Steven M. Seitz, Kevin TongACM Trans. Graphics 2021 · DOI · OpenReview

Abstract

We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.
BibTeX
@article{lawrence2021project,
  author = {Jason Lawrence and Dan B. Goldman and Supreeth Achar and Gregory Major Blascovich and Joseph G. Desloge and Tommy Fortes and Eric M. Gomez and Sascha Häberling and Hugues Hoppe and Andy Huibers and Claude Knaus and Brian Kuschak and Ricardo Martin-Brualla and Harris Nover and Andrew Ian Russell and Steven M. Seitz and Kevin Tong},
  title = {{Project starline: a high-fidelity telepresence system}},
  journal = {ACM Trans. Graphics},
  year = {2021},
  doi = {10.1145/3478513.3480490},
}

Loading related content...

Figure