Feel My Focus: Gaze-Triggered Digital Twin Creation for Remote Collaboration

Shih-Yu Ma, Suibi Che-Chuan Weng, Ming-Yuan Pai, Tom Yeh, Ellen Yi-Luen DoarXiv 2026 · DOI · arXiv

TL;DR

Feel My Focus transforms sustained visual attention from a simple pointer into a generative trigger for remote collaboration. By combining gaze-tracking glasses with one-shot 3D reconstruction, the system allows a user to instantly create and share a lightweight 3D "digital twin" of a real-world object with a remote partner in Mixed Reality (MR).

What Problem Does This Paper Address?

Remote collaboration tools have become excellent at transmitting voices and faces, but they still struggle to convey the physical context of a user's environment. If you want to show a remote partner a specific object—say, a unique artifact in a museum or a prototype on your desk—you are usually limited to pointing a camera at it.

While prior research has used eye-tracking to show a partner where someone is looking (referential gaze), it doesn't allow the partner to interact with that object. Traditional 3D reconstruction (photogrammetry) requires dozens of photos and minutes of processing, which kills the flow of a live conversation. This paper addresses the need for instant, intent-driven object sharing that prioritizes conversational responsiveness over perfect visual fidelity.

Methods

The authors developed an end-to-end pipeline that moves from a human glance to a shared 3D asset in seconds. The system architecture is modular, consisting of three main parts:

  1. Wearable Capture: A user wears GANZIN Sol Glasses, which track binocular eye movement and house a forward-facing camera. The system uses a two-stage "dwell" mechanism: a 250ms glance shows a visual cue, and if the user keeps looking for another 750ms, the system triggers a 3-second gaze-tracking window to generate a fixation heatmap.
  2. Server-Side Processing: The captured image and heatmap are sent to a server where:
  3. Segmentation & Recognition: A pipeline using SAM (Segment Anything Model) and YOLO isolates the object the user is looking at based on the gaze heatmap.
  4. One-Shot Reconstruction: The Real3D model generates a textured 3D mesh from that single segmented image.
  5. MR Visualization: The resulting digital twin is streamed via TCP to a remote user wearing a Meta Quest 3 headset. The remote user can then pick up, rotate, and discuss the virtual object as if it were physically present.
Fig. 2. Pipeline of Feel My Focus. A sustained gaze fixation from the Capture User is captured through gaze-tracking glasses and processed into a heatmap and image. The SAM/YOLO service segments the attended object and assigns a label, which is then reconstructed into a textured 3D mesh by Real3D. The resulting digital twin is transmitted to the MR User for immersive inspection and interaction.

Datasets

Not discussed in the paper. The system utilizes pre-trained models (SAM, YOLO, and Real3D) rather than training new models on a specific dataset.

Key Insights and Ideas

  • Gaze as a Generative Action: The core conceptual shift is treating attention as a command to create rather than just a command to point. This reframes eye-tracking from a passive awareness cue into an active creative tool.
  • Responsiveness > Fidelity: In collaborative settings, "good enough" 3D models delivered instantly are more valuable than photorealistic models that take five minutes to process. The authors intentionally chose one-shot reconstruction to maintain "experiential continuity."
  • The "Midas Touch" Problem: Using gaze as a trigger is high-stakes because humans look at many things unintentionally. The authors implemented a two-stage dwell and progress indicators to mitigate accidental captures, though they note that user-controlled confirmation steps are still a vital design consideration.

Key Results

The system was evaluated through a public demonstration with 35 participants.

  • Qualitative Feedback: Users consistently prioritized the immediacy of the sharing process over the visual quality of the 3D mesh. The ability to manipulate an object within a single conversational turn was seen as the system's primary strength.
  • Performance: The pipeline successfully instantiated shareable 3D proxies in situ. While reconstruction quality varied based on the object's geometry and texture, the generated meshes were consistently robust enough to serve as "conversational proxies."
Fig. 1. Overview of Feel My Focus. (A) The Capture User’s gaze highlights an attended object, which is detected and segmented. (B) The Capture User wearing gaze-tracking glasses focuses on a real-world item. (C) The MR User receives the reconstructed object in an immersive environment. (D) The digital twin appears in the MR scene for inspection and interaction.

Limitations and Future Work

  • Material Constraints: The current reconstruction pipeline struggles with glossy, transparent, or low-texture objects, which can lead to "degraded" meshes or ambiguous segmentation.
  • Robustness in Clutter: In environments with many overlapping objects, the system occasionally suffers from false gaze triggers or incorrect segmentation.
  • User Agency: While gaze-only triggering is seamless, it lacks the fine-grained control of manual selection. Future work will explore hybrid triggers (gaze + voice or hand gestures) to reduce unintended captures.
  • Scalability: The authors plan to characterize end-to-end latency and bandwidth requirements more rigorously to support deployment over standard Wi-Fi and 5G networks.

Summary updated

Abstract

We present Feel My Focus, a gaze-triggered mixed reality (MR) demonstration that enables rapid creation and sharing of lightweight 3D digital twins during remote collaboration. Unlike systems that use gaze primarily as a referential pointer, Feel My Focus treats sustained visual attention as a generative action: a fixation on a physical object initiates single-image capture, attention-guided segmentation, and one-shot 3D reconstruction, allowing a remote partner to inspect and discuss the object in situ. The system integrates gaze-tracking glasses, modular segmentation and reconstruction services, and an MR client for runtime streaming and interaction. This design prioritizes responsiveness and experiential continuity over photometric fidelity. We report observations from a public demonstration, highlighting design trade-offs, failure modes, and user control considerations for deploying gaze-triggered generative systems in real-world collaborative settings.
BibTeX
@misc{ma2026feel,
  author = {Shih-Yu Ma and Suibi Che-Chuan Weng and Ming-Yuan Pai and Tom Yeh and Ellen Yi-Luen Do},
  title = {{Feel My Focus: Gaze-Triggered Digital Twin Creation for Remote Collaboration}},
  year = {2026},
  eprint = {2611.11640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11640},
}
Figure