Mouse Trajectories and Gaze as Implicit Relevance Feedback

46 concepts32 papers2 proposals

When users interact with digital interfaces, traditional systems rely heavily on explicit feedback—such as clicks, likes, or star ratings—to infer what content is relevant. However, explicit feedback is sparse, introduces cognitive friction, and suffers from selection bias. For instance, as noted by Walber et al., standard input devices only capture the final selection result, completely discarding the rich contextual journey of how a user arrived at that decision.

To capture this hidden context, a powerful alternative paradigm has emerged: leveraging implicit behavioral signals as continuous, passive proxies for user attention and preference. By observing subconscious physiological and physical actions—specifically eye gaze and mouse cursor trajectories—systems can dynamically quantify attention and infer intent without interrupting the user's workflow.

Eye Gaze as a High-Bandwidth Window into Intent

The most direct physiological indicator of visual attention is eye tracking. When a user reads text or browses images, their eyes do not move smoothly; instead, they alternate between rapid jumps (saccades) and brief pauses (fixations). These micro-behaviors contain rich semantic information regarding cognitive load and information processing density.

Mapping these eye movements allows systems to capture user intent far more effectively than raw text alone. For example, Hardoon et al. demonstrated that analyzing eye-movement features like fixation duration and saccade length allows a regression framework to map gaze patterns to term importance weights. This enables the generation of "implicit queries" that rank unseen documents significantly better than standard text baselines. Similarly, Dengel et al. introduced the concept of "attentive documents" that record how content is consumed, using personalized gaze measures to rerank search results and expand queries.

Beyond text retrieval, eye gaze provides a powerful signal for visual and multimodal tasks. In image search, Pasupa et al. utilized implicit signals like fixation duration and gaze spread within a Tensor Ranking SVM to fuse visual features with gaze data, capturing nuances missed by global visual features. When navigating these visual spaces, specific behavioral patterns emerge: Auer et al. showed that "gaze regressions"—where a user's focus returns to a previously viewed image—serve as a much stronger indicator of relevance than simple dwell time. This implicit feedback loop can even be used as a passive supervision signal to learn semantic mappings between spoken words and on-screen objects in real time, as explored by Liu et al..

[User Reading Text/Images] 
       │
       ├─► Fixation Duration ──┐
       ├─► Saccade Length ─────┼─► [Implicit Relevance Feedback] ──► [Dynamic Re-ranking]
       └─► Gaze Regressions ───┘

This paradigm has recently extended to aligning artificial intelligence with human values. By analyzing eye movements during the reading of large language model (LLM) responses, López-Cardona et al. discovered that rejected responses require significantly more cognitive effort (measured via first fixation duration and total reading time) than preferred ones. Integrating these synthetic gaze features directly into reward modeling, as done in the GazeReward framework by Lopez-Cardona et al., yields substantial accuracy gains in predicting human preferences across conversational tasks.

The Scalability Leap: Mouse Trajectories as a Gaze Proxy

While eye tracking provides an incredibly rich, high-bandwidth signal, it historically suffered from a major limitation: it required specialized, expensive hardware and controlled laboratory environments. To overcome this, researchers looked for a ubiquitous software-based alternative.

The key insight, highlighted during the workshop summarized by Rodden et al., is that mouse cursor movements frequently correlate with eye-tracking data. Users often use their cursor as a physical marker or reading aid, making mouse tracking a highly scalable, low-cost proxy for visual attention.

By moving from server-side click logs to client-side cursor tracking, systems can capture user behavior even when no click occurs. For instance, Xu et al. demonstrated that monitoring user interaction time with search result summaries provides a critical pre-click indicator of intent, refining ranking models before a full page is even visited. White et al. integrated mouse cursor movements and scrolling behavior into a Dynamic Bayesian Network searcher model, allowing search engines to accurately predict which results users actually examined.

Traditional Logs:  [Query] ───────────────────────────────────────────────► [Click] (Sparse)
Cursor Tracking:   [Query] ──► [Horizontal Scan] ──► [Hesitation Hover] ──► [Click] (Dense)

What specific cursor patterns matter? Song et al. showed that incorporating fine-grained trajectory features—such as horizontal scanning and back-and-forth hesitation patterns—provides a granular representation of reading behavior and cognitive hesitation. This proves especially valuable for rare search queries where explicit click data is too sparse to build reliable relevance models.

This proxy mechanism works exceptionally well in non-linear layouts. In grid-based visual search interfaces, users often evaluate images without clicking them. To capture this, Mao et al. developed the Grid-based User Browsing Model (GUBM), which treats mouse hovers as high-frequency indicators of visual examination, successfully mapping the user's "Z-shaped" browsing patterns to estimate item relevance.

Translating Behavior to Quantifiable Attention

How do we mathematically bridge the gap between raw behavioral trajectories and actual visual attention? Modern approaches treat this as a sequence translation or machine learning problem.

For instance, Salvatori et al. developed AdSight, a Transformer-based sequence-to-sequence model that maps raw mouse cursor trajectories directly to fixation metrics across complex web layouts. By translating physical movement into predicted visual focus, it achieves near-second precision, matching the utility of hardware eye trackers at a fraction of the cost.

Furthermore, these behavioral patterns capture structural and semantic insights that transcend traditional feature engineering. In zero-shot image classification, Karessli et al. recorded how non-experts visually scan images to distinguish between classes, generating "gaze embeddings" that capture information processing density. These implicit embeddings actually outperformed text-based baselines and rivaled expensive, expert-annotated attributes.

Whether through the continuous monitoring of pervasive eye tracking discussed by Bednarik, the passive relevance classification frameworks of Barrett et al., or the high-frequency hover signals leveraged in exploratory search interfaces by He et al., the core mechanism remains the same. By treating subconscious physical behavior as a rich, continuous stream of implicit feedback, systems can align themselves more deeply with human intent, comprehension, and preference.

Key Foundational Work

Generating and using gaze-based document annotations (2008) — contributed 2 concept elements to this direction.
Information Retrieval by Inferring Implicit Queries from Eye Movements (2007) — contributed 2 concept elements to this direction.
Workshop on web information seeking and interaction (2007) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cursor_based_attention_proxy_e17b7500,
  title = {cursor-based attention proxy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cursor-based-attention-proxy/rdr_e17b7500d1db4309b8c95bc0494a1ea8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cursor_based_attention_proxy_e17b7500_v1,
  title = {cursor-based attention proxy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cursor-based-attention-proxy/rdr_e17b7500d1db4309b8c95bc0494a1ea8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake