TL;DR
This paper tackles the challenge of automatically identifying "Areas of Interest" (AOIs) in mobile eye-tracking data, a crucial but time-consuming manual task. They propose and evaluate two methods leveraging pre-trained deep learning models for image classification and object detection, showing promising results for AOIs with good concept matches but highlighting limitations with ambiguous or unsupported concepts.
What Problem Does This Paper Address?
Understanding where people look is fundamental to many research fields, from human-computer interaction to cognitive science. Eye-tracking technology captures this gaze data, and researchers often analyze it by defining "Areas of Interest" (AOIs) – specific regions within a visual scene that are hypothesized to be important. The process of manually annotating these AOIs, especially for dynamic video stimuli or mobile eye-tracking where each recording is unique, is incredibly tedious and subjective. This paper aims to automate this AOI detection process, making eye-tracking research more efficient and objective, particularly for mobile setups.
Methods
The authors propose two primary methods for automatic visual attention detection:
- Gaze-Guided Image Classification (IC): This method involves cropping image patches around the detected gaze points. These patches are then fed into a pre-trained image classification model (like ResNet) to predict the object class. The system aggregates probabilities for similar classes and uses a working memory and threshold-based approach to identify sustained attention events (start/end of attention).
- Object Detection (OD): This approach uses a pre-trained object detection model (like Mask R-CNN) to identify objects within video frames. Fixation events from the eye tracker are then matched with these detected object regions. If a fixation falls within an object's bounding box or mask, an attention event is registered.
Both methods leverage pre-trained models, meaning they don't require extensive new training data for common object categories. The paper focuses on exploring the effectiveness of these off-the-shelf models.
Datasets
The primary dataset used for evaluation is the VISUS dataset. This dataset contains eye-tracking data from 25 participants across 11 video stimuli, with manual annotations for AOIs. The authors treat the fixation events that hit these manually defined AOI bounding boxes as the ground truth.
Key Insights and Ideas
- Pre-trained models are powerful but have limits: The study demonstrates that pre-trained computer vision models can effectively detect attention for AOIs that have clear, well-defined concepts matching the model's training data (e.g., "car").
- Concept ambiguity is a major hurdle: When AOIs represent concepts not directly present in the pre-trained model's classes, or when multiple AOIs share the same concept (e.g., different types of "person"), performance significantly degrades. This is a key challenge for real-world applications.
- Annotation quality matters: The paper highlights that coarse or overly broad manual AOI annotations can lead to a high number of "deletion errors" (false negatives) because the automated methods, which often produce more precise object boundaries, fail to match these loose ground truth regions.
- Object detection offers better recall, classification offers better precision: OD-bbox generally achieved higher recall (detecting more true attention events), while IC methods tended to have higher precision (fewer false positives), especially when concepts were well-matched.
Key Results
- Failing AOIs: The methods failed to detect attention for a significant number of AOIs, particularly those without direct mappings to ImageNet or MS COCO classes. This accounted for 82.35% of AOIs for IC and 73.53% for OD.
- Overall Performance: For the AOIs that were supported, OD-bbox achieved the best progressive recall (61.06%), while IC methods offered better progressive precision (94.33%).
- Error Analysis: Deletions (false negatives) were the most frequent error across all methods. Insertions (false positives) were more prevalent in OD methods.
- Impact of Re-annotation: Re-annotating the ground truth with tighter, more precise object boundaries significantly reduced deletion errors and improved recall for both methods.
Limitations and Future Work
The authors acknowledge several limitations:
- Unsupported Concepts: The reliance on pre-trained models means performance is heavily dependent on the availability of matching classes.
- Ambiguous Concepts (Challenge II): Differentiating between AOIs that share the same concept (e.g., multiple "person" instances) remains a challenge.
- Gaze Estimation Errors: Inaccurate gaze data can lead to missed detections, especially when combined with precise object detection.
- Annotation Granularity: The VISUS dataset's broad annotations contributed to false negatives.
Future work suggested includes:
- Interactive Machine Learning: Adapting models to new scenarios or unsupported AOIs through user feedback and fine-tuning.
- Multi-Object Tracking: Using advanced tracking to differentiate between instances of the same concept and handle AOIs with similar appearances.
- Error-Aware Gaze-to-Object Mapping: Developing methods that can compensate for gaze estimation errors.
- Improved Fixation Detection: Investigating better fixation detection algorithms, especially for mobile eye tracking.
Summary updated