Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-Engaging in Immersive VR Meetings

Geonsun Lee, Yue Yang, Jennifer Healey, Dinesh Manocha

Introduction

In immersive Virtual Reality (VR) environments, maintaining a sense of "presence" is critical for effective collaboration. However, users frequently experience "dropouts"—brief departures from the virtual space due to physical world distractions, technical issues, or multitasking. Re-entering a fast-paced VR meeting after a dropout is often socially awkward and cognitively demanding. Users must either interrupt the flow to ask for a recap or attempt to parse lengthy, static transcription logs that distract from the social cues of the group.

To address this, researchers from the University of Maryland, Stanford, and Adobe Research developed EngageSync, a context-aware transcription system. EngageSync utilizes Large Language Models (LLMs) and gaze-tracking to provide on-demand summaries anchored to participant avatars, facilitating seamless re-engagement without sacrificing social presence.

The EngageSync System

EngageSync operates in two primary modes to balance information utility with social presence:

  1. Engagement Mode: For active participants, the system provides "on-demand" information. By gazing at a speaker and performing a pinch gesture, users can view a live transcription. Gazing at a listener provides a brief summary of their last contribution.
  2. Re-engagement Mode: Triggered when a user returns from a dropout, this mode automatically generates 15-word summaries of missed dialogue for each participant. These summaries are spatially anchored above the respective avatars and are dismissed after a 1.5-second gaze, ensuring the user has processed the information before returning to the live discussion.
Overview of EngageSync’s Engagement and Re-engagement modes, illustrating the transition between live transcription and dropout summaries.

The system is built in Unity, leveraging the Meta Movement SDK for gaze tracking and GPT-4-Turbo for real-time summarization. A host-mode server ensures that transcriptions and summaries are synchronized across all participants.

User Evaluation

The researchers conducted two user studies to evaluate EngageSync:

  • Formative Study (n=12): Established that users prefer avatar-anchored text over traditional table-fixed logs, which were found to be overwhelming and disruptive to eye contact.
  • Main User Study (n=30): Compared EngageSync against two baselines: Table-fixed (TableTI) and Always-on Avatar-fixed (AvatarTI) transcriptions. Participants engaged in debates in both small (3 avatars) and mid-sized (7 avatars) group settings.
The VR meeting environment in a mid-sized group setting, demonstrating how transcription panels are spatially anchored above avatars to aid memory and social presence.

Key Findings

The study demonstrated that EngageSync significantly outperforms traditional methods in several key areas:

  • Enhanced Social Presence: By anchoring text to avatars, users spent more time looking at their colleagues rather than at a central UI element. This led to higher co-presence and attentional allocation scores ($p < .05$).
  • Improved Information Recall: In larger groups, EngageSync users achieved higher quiz scores regarding meeting content compared to the AvatarTI baseline. Spatial anchoring helped users associate specific points with the correct speakers, particularly for less talkative participants.
  • Efficient Re-engagement: The use of LLM-generated summaries allowed users to catch up significantly faster than scrolling through full transcripts, reducing the cognitive load of re-entering the conversation.

Conclusion and Future Work

EngageSync demonstrates that context-aware, spatially anchored summaries can bridge the gap between information needs and social presence in VR. While the system showed strong results, future work will address UI placement in complex environments and the impact of LLM latency on spontaneous, high-speed interactions.

Citation: Geonsun Lee, Yue Yang, Jennifer Healey, and Dinesh Manocha.

  1. Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-engaging in Immersive VR Meetings. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’25).

Summary updated

BibTeX
@inproceedings{lee2025since,
  author = {Geonsun Lee and Yue Yang and Jennifer Healey and Dinesh Manocha},
  title = {{Since U Been Gone: Augmenting Context-Aware Transcriptions for Re-Engaging in Immersive VR Meetings}},
  booktitle = {CHI},
  year = {2025},
  doi = {10.1145/3706598.3714078},
}
Figure