Introduction
As Augmented Reality (AR) technology matures, teleconferencing has emerged as a primary use case. However, current implementations often force a compromise between visual realism and spatial presence. Conventional 2D video windows, while photorealistic, feel detached from the local environment. Conversely, 3D virtual avatars provide a sense of shared space but often lack the subtle facial expressions of real humans or fall into the "uncanny valley." While high-fidelity volumetric capture systems like Google’s Project Starline offer a solution, they require specialized, non-mobile hardware. This paper investigates life-size 2D video avatars as a mobile and effective "middle ground" for HMD-based AR teleconferencing.
System Design and Implementation
The researchers developed a prototype using a standard webcam and a Microsoft HoloLens 2. To create the video avatars, the system employs the MobileNetV3 model for real-time background removal, isolating the user from their physical surroundings. These isolated video feeds are then rendered as life-size "billboards" within the local user's AR view. To maintain a natural conversational feel, these billboards are programmed to always face the local user, regardless of their movement.
Experimental Methodology
The study was divided into three phases to evaluate the effectiveness and spatial requirements of video avatars:
- Pilot Study: 18 participants helped determine the optimal spatial configuration for groups of 1 to 4 remote users. This involved defining the "Radius" (the distance between the local user and the remote participants) and the "Radian" (the angular spread between participants).
- Comparative Evaluation: 20 participants compared the proposed Video Avatar mode against three baselines: Traditional Video (PC-based), Video Grid (a floating 2D screen in AR), and Virtual Avatar (cartoonish 3D models).
- Field of View (FoV) Exploration: Using a VR headset (HTC Vive Pro) to simulate AR environments, the researchers tested how varying FoVs (from 30° to 110°) influenced where users preferred to place remote participants.
Key Findings
Realism vs. Presence
The evaluation study revealed that Video Avatars successfully bridge the gap between fidelity and co-presence. Participants rated the Video Avatar mode as having visual fidelity comparable to traditional video (mean 5.1/7) and a sense of co-presence comparable to 3D virtual avatars (mean 5.2/7). It significantly outperformed traditional video in co-presence and virtual avatars in fidelity.
The Impact of Hardware Constraints
A significant contribution of this work is the analysis of how hardware limitations, specifically the Field of View (FoV) of AR headsets, affect social interaction. The researchers found a strong positive correlation between FoV and the preferred Radian (horizontal spread). In headsets with a narrow FoV, users instinctively cluster remote participants closer together to keep them within the visible frame without requiring excessive head rotation.
Interestingly, the preferred Radius (distance) was less sensitive to FoV. Users generally maintained a distance that allowed them to see the remote participant's full body or upper torso, prioritizing natural social cues over keeping the entire avatar within the FoV at all times.
Predictive Modeling for AR Developers
To assist future AR development, the authors derived regression models to automate avatar placement. For example, the optimal horizontal spread for two remote users can be calculated as: $$Radian_{2RU} = -0.0012FoV^2 + 0.45FoV + 20.20$$ These models allow systems to dynamically adjust the layout of a teleconference based on the specific hardware capabilities of the user's HMD.
Limitations and Future Directions
While promising, the 2D billboard approach has limitations. Significant lateral movement by the local user can reveal the "flatness" of the avatar. Future research could explore neural rendering techniques (such as NeRFs) to synthesize 3D views from 2D feeds. Additionally, automatic relighting is necessary to ensure the avatars' lighting matches the local physical environment, further enhancing immersion. Finally, while this study focused on standing interactions among students, future work should investigate more diverse demographics and varied social contexts, such as seated meetings.
Conclusion
Life-size 2D video avatars provide a high-fidelity, high-presence solution for AR teleconferencing that is compatible with current mobile hardware. By understanding the interplay between social proxemics and hardware constraints like Field of View, developers can create more immersive and natural remote collaboration experiences.
Summary updated