TL;DR
Feel My Focus transforms sustained visual attention from a simple pointer into a generative trigger for remote collaboration. By combining gaze-tracking glasses with one-shot 3D reconstruction, the system allows a user to instantly create and share a lightweight 3D "digital twin" of a real-world object with a remote partner in Mixed Reality (MR).
What Problem Does This Paper Address?
Remote collaboration tools have become excellent at transmitting voices and faces, but they still struggle to convey the physical context of a user's environment. If you want to show a remote partner a specific object—say, a unique artifact in a museum or a prototype on your desk—you are usually limited to pointing a camera at it.
While prior research has used eye-tracking to show a partner where someone is looking (referential gaze), it doesn't allow the partner to interact with that object. Traditional 3D reconstruction (photogrammetry) requires dozens of photos and minutes of processing, which kills the flow of a live conversation. This paper addresses the need for instant, intent-driven object sharing that prioritizes conversational responsiveness over perfect visual fidelity.
Methods
The authors developed an end-to-end pipeline that moves from a human glance to a shared 3D asset in seconds. The system architecture is modular, consisting of three main parts:
- Wearable Capture: A user wears GANZIN Sol Glasses, which track binocular eye movement and house a forward-facing camera. The system uses a two-stage "dwell" mechanism: a 250ms glance shows a visual cue, and if the user keeps looking for another 750ms, the system triggers a 3-second gaze-tracking window to generate a fixation heatmap.
- Server-Side Processing: The captured image and heatmap are sent to a server where:
- Segmentation & Recognition: A pipeline using SAM (Segment Anything Model) and YOLO isolates the object the user is looking at based on the gaze heatmap.
- One-Shot Reconstruction: The Real3D model generates a textured 3D mesh from that single segmented image.
- MR Visualization: The resulting digital twin is streamed via TCP to a remote user wearing a Meta Quest 3 headset. The remote user can then pick up, rotate, and discuss the virtual object as if it were physically present.
Datasets
Not discussed in the paper. The system utilizes pre-trained models (SAM, YOLO, and Real3D) rather than training new models on a specific dataset.
Key Insights and Ideas
- Gaze as a Generative Action: The core conceptual shift is treating attention as a command to create rather than just a command to point. This reframes eye-tracking from a passive awareness cue into an active creative tool.
- Responsiveness > Fidelity: In collaborative settings, "good enough" 3D models delivered instantly are more valuable than photorealistic models that take five minutes to process. The authors intentionally chose one-shot reconstruction to maintain "experiential continuity."
- The "Midas Touch" Problem: Using gaze as a trigger is high-stakes because humans look at many things unintentionally. The authors implemented a two-stage dwell and progress indicators to mitigate accidental captures, though they note that user-controlled confirmation steps are still a vital design consideration.
Key Results
The system was evaluated through a public demonstration with 35 participants.
- Qualitative Feedback: Users consistently prioritized the immediacy of the sharing process over the visual quality of the 3D mesh. The ability to manipulate an object within a single conversational turn was seen as the system's primary strength.
- Performance: The pipeline successfully instantiated shareable 3D proxies in situ. While reconstruction quality varied based on the object's geometry and texture, the generated meshes were consistently robust enough to serve as "conversational proxies."
Limitations and Future Work
- Material Constraints: The current reconstruction pipeline struggles with glossy, transparent, or low-texture objects, which can lead to "degraded" meshes or ambiguous segmentation.
- Robustness in Clutter: In environments with many overlapping objects, the system occasionally suffers from false gaze triggers or incorrect segmentation.
- User Agency: While gaze-only triggering is seamless, it lacks the fine-grained control of manual selection. Future work will explore hybrid triggers (gaze + voice or hand gestures) to reduce unintended captures.
- Scalability: The authors plan to characterize end-to-end latency and bandwidth requirements more rigorously to support deployment over standard Wi-Fi and 5G networks.
Summary updated