Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing

Xuanyu Wang, Weizhan Zhang, Christian Sandor, Hongbo Fu

Introduction

Current Augmented Reality (AR) teleconferencing faces a significant "fidelity vs. co-presence" trade-off. High-fidelity 3D reconstruction remains prohibitively expensive and complex for consumer use. Conversely, mobile-friendly solutions often rely on cartoonish virtual avatars that lack realism or "video grids"—floating 2D boxes that feel detached from the user's physical environment.

This research proposes a middle ground: life-size 2D video avatars. By using background-removed video streams rendered at a 1:1 scale within the user's local space, the system aims to provide the photorealism of a video call with the spatial presence of a 3D avatar.

The proposed system uses background-removed 2D video avatars (left) and a geometric model (right) to define social placement via Radius and Radian.

Methodology

The researchers investigated how hardware constraints, specifically the limited Field of View (FoV) of current Head-Mounted Displays (HMDs), influence social proxemics. The study followed a three-stage approach:

  1. Pilot Study: Using a Microsoft HoloLens 2, 18 participants manually adjusted the placement (Radius and Radian) of up to four remote video avatars to find the most comfortable configuration.
  2. System Prototype: A functional AR teleconferencing system was built using MediaPipe for real-time background removal and pose estimation. This was compared against traditional 2D video (PC), 3D cartoon avatars, and floating video grids.
  3. VR-Simulated AR Study: To anticipate future hardware, the team used an HTC Vive Pro to simulate AR environments with varying FoVs, ranging from 10° to a human-like 180°.

Key Insights: FoV and Social Geometry

The study revealed that a headset's FoV significantly dictates where users want remote participants to "stand."

  • The "Bunching" Effect: In headsets with a narrow FoV, users tend to place remote participants closer together (smaller Radian) to keep them within the visible window.
  • The "Step Back" Effect: When vertical FoV is limited, users prefer to move avatars further away (larger Radius). This allows them to see the remote person's full body without constant vertical head movement.
  • Fidelity vs. Co-presence: Video avatars were the most preferred mode. They maintained high fidelity scores (~5.1/7) similar to traditional video, while achieving co-presence scores (~5.2/7) nearly equal to 3D avatars (~5.3/7), far exceeding the performance of floating video boxes (~2.9/7).
A simulation of the limited Field of View (FoV) window in an AR environment, illustrating how hardware constraints affect the visibility of remote participants.

Mathematical Modeling for Auto-Placement

To assist developers, the authors derived regression models to automatically calculate optimal avatar placement based on headset specifications. For example, the model for two remote users (RU) is defined as:

$$Radian_{2RU} = -0.0012FoV^2 + 0.45FoV + 20.20$$

Conclusion and Future Work

Life-size 2D video avatars offer a compelling, low-cost alternative for AR teleconferencing that balances realism and spatial presence. While the current "billboard" approach breaks the 3D illusion if a user walks around the avatar, future work involving neural rendering (e.g., NeRFs) and automatic lighting estimation could further bridge the gap between 2D video and true 3D presence. The study successfully demonstrates that VR simulations are a reliable method for predicting user behavior for future AR hardware iterations.

Summary updated

Abstract

Augmented Reality (AR) teleconferencing allows spatially distributed users to interact with each other in 3D through agents in their own physical environments. Existing methods leveraging volumetric capturing and reconstruction can provide a high-fidelity experience but are often too complex and expensive for everyday use. Other solutions target mobile and effortless-to-setup teleconferencing on AR Head Mounted Displays (HMD). They directly transplant the conventional video conferencing onto an AR-HMD platform or use avatars to represent remote participants. However, they can only support either a high fidelity or a high level of co-presence. Moreover, the limited Field of View (FoV) of HMDs could further degrade users’ immersive experience. To achieve a balance between fidelity and co-presence, we explore using life-size 2D video-based avatars (video avatars for short) in AR teleconferencing. Specifically, with the potential effect of FoV on users’ perception of proximity, we first conducted a pilot study to explore the local-user-centered optimal placement of video avatars in small-group AR conversations. With the placement results, we then implement a proof-of-concept prototype of video-avatar-based teleconferencing. We conduct user evaluations with our prototype to verify its effectiveness in balancing fidelity and co-presence. Following the indication in the pilot study, we further quantitatively explore the effect of FoV size on the video avatar’s optimal placement through a user study involving more FoV conditions in a VR-simulated environment. We regress placement models to serve as references for computationally determining video avatar placements in such teleconferencing applications on various existing AR HMDs and future ones with bigger FoVs.
BibTeX
@article{wang2025real,
  author = {Xuanyu Wang and Weizhan Zhang and Christian Sandor and Hongbo Fu},
  title = {{Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  doi = {10.1109/tvcg.2024.3466554},
}
Figure