Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras

Hanzhang Tu, Ruizhi Shao, Xue Dong, Shunyuan Zheng, Hao Zhang, Lili Chen, Meili Wang, Wenyu Li, Siyan Ma, Shengping Zhang, Boyao Zhou, Yebin Liu

Introduction

Immersive telepresence—the sensation of "co-presence" where users feel as though they are sharing the same physical space—has long been a primary goal of communication technology. While industry leaders have produced impressive prototypes like Google’s Project Starline, these systems typically rely on expensive hardware, including dozens of cameras, active depth sensors (Time-of-Flight or structured light), and massive compute clusters. These requirements make high-fidelity telepresence inaccessible to the general public and enterprise environments with limited budgets.

Tele-Aloha introduces a paradigm shift by achieving "Starline-quality" 3D communication using a sparse setup: just four standard 4K RGB cameras and a single consumer-grade NVIDIA RTX 4090 GPU. By replacing hardware-heavy depth sensing with a novel neural rendering pipeline, Tele-Aloha delivers 2K resolution at 30 FPS for approximately $15,000.

The Challenge of Sparse Viewpoints

The primary technical hurdle in low-budget telepresence is the "sparse view" problem. When cameras are placed far apart to cover a wide field of view, the resulting parallax—the difference in perspective between cameras—is significant. Traditional depth sensors often struggle with real-world complexities like dark clothing or reflective surfaces, and standard stereo-matching algorithms frequently fail when the baseline between cameras is too wide.

Tele-Aloha addresses these challenges through a three-stage "Efficient Novel View Synthesis" algorithm designed to maintain high fidelity even when visual data is limited.

Technical Methodology

1. Cascaded Disparity Estimation

To establish a reliable 3D proxy of the user, the system employs a "narrow-to-wide" strategy. It utilizes two pairs of cameras. A narrow-baseline pair (cameras positioned close together) provides a stable initial depth estimation. This initial "guess" serves as a foundation for the wide-baseline pair, which captures a broader perspective. This cascaded approach significantly reduces depth errors and temporal jitter, which are common in sparse camera arrays.

2. Generalizable 3D Gaussian Splatting (3DGS)

Unlike traditional 3DGS, which requires lengthy per-scene training, Tele-Aloha utilizes a generalizable version. The system extracts high-dimensional "latent features" from 2D images and "splats" them into 3D space. By splatting features rather than raw RGB values, the system becomes more resilient to depth inaccuracies. A 2D UNet decoder then processes these features to "paint" the final image, effectively filling in holes caused by self-occlusion (e.g., an arm blocking the view of the torso).

3. Occlusion-aware Rendering Refinement

Rendering high-resolution 3D content in real-time is computationally expensive. Tele-Aloha optimizes this by rendering the initial neural output at 1024x1024. To reach a final 2K resolution, a lightweight refiner network blends this neural output with high-frequency details warped directly from the original 4K camera feeds. This hybrid approach ensures that fine details—such as fabric textures and skin pores—are preserved without overtaxing the GPU.

Evaluation and Results

Performance and Latency

The system achieves a consistent 30 FPS with an end-to-end latency of 149ms. This latency is well within the acceptable threshold for natural, interactive conversation, ensuring that the 3D effect does not come at the cost of communication fluidity.

Quantitative and Qualitative Quality

The model was trained and evaluated using the THuman-Sit and THuman2.0 datasets, comprising thousands of high-quality human scans. In comparative testing, Tele-Aloha outperformed existing state-of-the-art systems (such as FloRen and GPS-Gaussian) in PSNR and SSIM metrics. Qualitatively, the system excels in "occlusion zones," providing sharper and more stable reconstructions during movement compared to previous methods.

Cost Efficiency

By utilizing consumer-grade hardware and an autostereoscopic (glasses-free 3D) display, the total hardware cost is roughly $15,000. This represents a significant reduction in cost compared to commercial telepresence booths, which can cost upwards of six figures.

Limitations and Future Work

While Tele-Aloha represents a major step forward, certain challenges remain:

  • Specular Surfaces: Highly reflective objects, such as eyeglasses or glass containers, can still cause artifacts in the disparity estimation.
  • Background Matting: The system's quality is partially dependent on the accuracy of background removal. Imperfect matting can lead to background "bleeding" into the 3D reconstruction.
  • Material Anisotropy: Future iterations aim to better handle view-dependent materials, such as shiny fabrics, to further enhance realism.

Conclusion

Tele-Aloha demonstrates that high-authenticity 3D telepresence does not require an enterprise-grade budget. By leveraging cascaded depth estimation and generalizable neural rendering, the system provides a scalable blueprint for the future of immersive digital communication.

Summary updated

BibTeX
@inproceedings{tu2024tele,
  author = {Hanzhang Tu and Ruizhi Shao and Xue Dong and Shunyuan Zheng and Hao Zhang and Lili Chen and Meili Wang and Wenyu Li and Siyan Ma and Shengping Zhang and Boyao Zhou and Yebin Liu},
  title = {{Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657491},
}
Figure