Decoupling Target Selection and Execution with Eye Gaze

24 concepts22 papers1 proposal

When we interact with a digital interface, our eyes naturally dart toward our target long before our hands move to click, tap, or point. Traditional user interfaces bind both spatial localization and command execution to a single input channel, such as a mouse cursor or a finger tap. This creates a fundamental tension: eye gaze is incredibly fast at scanning but prone to jitter and accidental activations—a phenomenon known as the "Midas Touch" problem.

To resolve this, researchers have converged on a coarse-to-fine multimodal paradigm. By decoupling the interaction into two distinct phases—using eye or head gaze for rapid, coarse target localization (the where) and a secondary, high-precision input modality for confirmation or fine manipulation (the when or how)—interfaces can achieve both extreme speed and robust accuracy.

The Division of Labor: Global vs. Local Control

Human visual attention naturally operates at a global scale, while physical manipulation operates locally. Offloading global scanning to gaze allows manual inputs to focus entirely on local control, preventing the two modalities from remaining inefficiently tethered, as explored by Patrick Taylor et al. in Human Strategies for Multitasking, Search, and Control Improved via Real-Time Memory Aid for Gaze Location.

This complementary nature is behaviorally evident during complex tasks like robotic teleoperation. Henny Admoni et al. demonstrated in Gaze Complements Control Input for Goal Prediction During Assisted Teleoperation that natural gaze provides high information gain regarding a user's spatial intent early in a task, whereas manual input provides the necessary temporal precision as the cost of error increases. Fusing these parallel streams allows systems to predict goals faster and reduce physical operator fatigue. Similarly, Animesh Garg et al. in Fast Explicit-Input Assistance for Teleoperation in Clutter highlight that aligning a robot's assistance directly with the operator's visual focus minimizes the cognitive overhead of switching between input modalities.

Defeating the Midas Touch

The primary barrier to gaze-only interfaces is that looking cannot be equated with clicking. In Gaze-Based Menu Navigation in Virtual Reality: A Comparative Study of Layouts and Interaction Techniques, László Kopácsi et al. found that users strongly prefer a multimodal "gaze-to-point, button-to-click" strategy over gaze-only dwell times. This physical gating mechanism provides tactile confirmation and eliminates artificial selection delays.

This separation of concerns is highly versatile across physical and virtual environments:

Coarse-to-Fine Optimization in Practice

Consider a large smartphone screen where reaching the top corners with a thumb is ergonomically taxing. In GazeSwipe: Enhancing Mobile Touchscreen Reachability through Seamless Gaze and Finger-Swipe Integration, Zhuojiang Cai et al. use eye gaze to instantly snap a cursor to a coarse target area, and then use a short, localized thumb swipe for fine-grained adjustment and confirmation. This reduces thumb travel by more than half, though it introduces a minor latency trade-off due to the multi-step process of gazing, freezing, and swiping.

[ Visual Exploration ] ---> [ Coarse Target Acquisition ] ---> [ Fine Confirmation / Manipulation ]
      (Eye Gaze)                    (Gaze Snap)                     (Touch, Voice, or Click)

In virtual reality (VR) and spatial computing, this paradigm optimizes workflow efficiency across infinite workspaces. Verena Biener et al. (Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers) demonstrated that automated input redirection based on eye-gaze is far more efficient than manual bimanual selection for moving content across multiple virtual windows. Furthermore, Travis Gesslein et al. (Pen-based Interaction with Spreadsheets in Mobile Virtual Reality) combined head- or eye-gaze for initial target identification with tactile gestures on a physical tablet boundary for confirmation, significantly reducing physical fatigue.

This division of labor extends naturally to other modalities like voice and neural control:

Designing the Interaction Flow

To build effective hybrid interfaces, designers must map inputs to preserve immersion and flow. As Momona Yamagami et al. note in Two-In-One: A Design Space for Mapping Unimanual Input into Bimanual Interactions in VR for Users with Limited Movement, mapping secondary virtual actions to alternative channels like gaze preserves interaction flow much better than forcing temporal mode-switching on a single device.

Understanding the transition from visual exploration to final decision-making requires modeling how these modalities interlock. By analyzing high-frequency eye-tracking data alongside cursor paths and click events, the authors of RecGaze: The First Eye Tracking and User Interaction Dataset for Carousel Interfaces (Santiago de Leon-Martinez et al.) provided a foundational framework for capturing this cognitive pipeline.

Ultimately, as spatial computing evolves, Zhimin Wang et al. (Towards spatial computing: recent advances in multimodal natural interaction for Extended Reality headsets) emphasize that assigning high-speed, coarse-grained selection to eye-gaze and high-precision manipulation to hand gestures represents the most natural, fatigue-reducing paradigm for spatial environments. Even in assistive contexts, such as the E-Gaze glasses developed by Shi Qiu 0001 et al. (HCI Design for People with Visual Disability in Social Interaction), combining multi-modal feedback loops is crucial for bridging the gap between perception and execution.

Key Foundational Work

Eye drop: an interaction concept for gaze-supported point-to-point content transfer (2013) — contributed 2 concept elements to this direction.
Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze and Touch (2013) — contributed 2 concept elements to this direction.
Vision-based face tracking system for window interface (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gaze_directed_selection_1eb3466c,
  title = {gaze-directed selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaze-directed-selection/rdr_1eb3466c83e948cdbc0ef26db6dc6619},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_gaze_directed_selection_1eb3466c_v1,
  title = {gaze-directed selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaze-directed-selection/rdr_1eb3466c83e948cdbc0ef26db6dc6619/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake