eyeNotate: Interactive Annotation of Mobile Eye Tracking Data Based on Few-Shot Image Classification

Michael Barz, Omair Shahzad Bhatti, Hasan Md Tusfiqur Alam, Duy Minh Ho Nguyen, Kristin Altmeyer, Sarah Malone, Daniel SonntagJournal of Eye Movement Research 2025 · DOI

TL;DR

This paper introduces eyeNotate, a web-based tool for semi-automatically annotating mobile eye-tracking data. By leveraging few-shot image classification and interactive machine learning, eyeNotate aims to significantly speed up the laborious process of mapping eye fixations to areas of interest (AOIs), making mobile eye-tracking research more efficient.

What Problem Does This Paper Address?

Analyzing data from mobile eye-tracking devices, which capture an egocentric view of the user's environment, is a critical but incredibly time-consuming task. Researchers need to map each eye fixation to specific "areas of interest" (AOIs) within the scene to understand visual attention. This process is manual and unique for each participant because their viewpoints differ, making it a major bottleneck in research and the development of attention-aware interfaces. Existing automated methods often struggle with the variability of mobile eye-tracking data or lack flexibility.

Methods

The core of eyeNotate is a two-pronged approach:

  • Baseline Annotation Tool: A user-friendly interface that mimics video editing software, allowing users to manually assign fixations to AOIs. This serves as the benchmark for comparison.
  • IML-Support Version: This version integrates a few-shot image classification (FSL) model, specifically the Feature Map Reconstruction Network (FRNet). This model generates AOI suggestions for each fixation. Crucially, it employs an interactive machine learning (IML) paradigm: user feedback (accepting, rejecting, or correcting suggestions) is used to continuously retrain and improve the FSL model. This allows the model to adapt to the specific target domain and user preferences.

Datasets

The study utilized an existing mobile eye-tracking dataset (n=48) collected at Saarland University for an educational research scenario involving electrical circuits. For the case study, a subset of data from a single individual was used. For the post-hoc machine learning experiment, the remaining data from 47 individuals was used.

Key Insights and Ideas

  • FSL for AOI Mapping is Promising: Even with limited training data, few-shot learning shows potential for generating useful AOI suggestions, especially when combined with user feedback.
  • User Trust is Paramount: The effectiveness of IML-support hinges on user trust in the model's suggestions. Low perceived model performance, even if inaccurate, can lead users to revert to manual annotation, negating the efficiency gains.
  • Similar Appearances are a Major Hurdle: Differentiating between AOIs with similar visual characteristics (e.g., left vs. right tablet) is a significant challenge for current models, impacting both accuracy and user confidence.
  • Clean UI Design is Crucial: Participants appreciated the intuitive and clean design of eyeNotate, highlighting its usability and learnability, regardless of the IML features.

Key Results

  • Validity and Reliability: Both the baseline and IML-support versions achieved high annotation accuracy (around 95%) and excellent inter-rater reliability (Fleiss's κ ≥ 0.9).
  • Efficiency: Contrary to expectations, the IML-support version did not significantly improve efficiency. In fact, annotators were slightly slower on average (3.44% slower). This was attributed to low user trust in the model's suggestions, leading them to manually verify or correct most suggestions.
  • Usability: The baseline version was rated "excellent" for usability. The IML-support version showed higher variance, with some users rating it "poor" due to perceived model inaccuracies and confusion.
  • ML Performance: The FRNet model achieved the best performance in the post-hoc experiment, with an accuracy of 58.78% in the final setting. However, even the best models struggled with class imbalance and similar-looking AOIs, achieving relatively low F1 scores for most specific AOIs.

Limitations and Future Work

  • Small Sample Size: The case study involved only three trained annotators, limiting the generalizability of the findings. Future work should include a larger and more diverse participant pool, including lay users.
  • Model Performance: The primary limitation identified was the constrained performance of the FSL model, particularly with visually similar AOIs. This led to low user trust and limited efficiency gains.
  • Future Directions: The authors suggest exploring more sophisticated computer vision methods like multi-object tracking, 3D scene reconstruction, and graph neural networks to better handle the complexity of mobile eye-tracking data. Investigating meta-models that selectively offer suggestions for classes the model performs well on, or simplifying the classification problem (e.g., to a four-class problem), are also promising avenues. Additionally, exploring how lay users interact with and benefit from such tools is recommended.

Summary updated

Abstract

Mobile eye tracking is an important tool in psychology and human-centered interaction design for understanding how people process visual scenes and user interfaces. However, analyzing recordings from head-mounted eye trackers, which typically include an egocentric video of the scene and a gaze signal, is a time-consuming and largely manual process. To address this challenge, we develop eyeNotate, a web-based annotation tool that enables semi-automatic data annotation and learns to improve from corrective user feedback. Users can manually map fixation events to areas of interest (AOIs) in a video-editing-style interface (baseline version). Further, our tool can generate fixation-to-AOI mapping suggestions based on a few-shot image classification model (IML-support version). We conduct an expert study with trained annotators (n = 3) to compare the baseline and IML-support versions. We measure the perceived usability, annotations' validity and reliability, and efficiency during a data annotation task. We asked our participants to re-annotate data from a single individual using an existing dataset (n = 48). Further, we conducted a semi-structured interview to understand how participants used the provided IML features and assessed our design decisions. In a post hoc experiment, we investigate the performance of three image classification models in annotating data of the remaining 47 individuals.
BibTeX
@article{barz2025eyenotate,
  author = {Michael Barz and Omair Shahzad Bhatti and Hasan Md Tusfiqur Alam and Duy Minh Ho Nguyen and Kristin Altmeyer and Sarah Malone and Daniel Sonntag},
  title = {{eyeNotate: Interactive Annotation of Mobile Eye Tracking Data Based on Few-Shot Image Classification}},
  journal = {Journal of Eye Movement Research},
  year = {2025},
  doi = {10.3390/jemr18040027},
}
Figure