Abstract
In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research.
Note: This article has been retracted by the publisher. Readers should consult the official retraction notice for details.
Introduction: The Reporting Gap in Eye Tracking
Eye tracking has become a fundamental tool across diverse scientific disciplines, from cognitive psychology to human-computer interaction. However, the field currently faces a significant challenge: inconsistent reporting practices. This lack of transparency hinders reproducibility and makes it difficult to compare results across different studies. While some guidelines exist, they are often based on expert consensus rather than a systematic review of empirical evidence.
This research addresses this "replication crisis" by synthesizing decades of empirical work to identify the factors that truly impact data quality. The goal is to establish a robust, evidence-based foundation for reporting eye-tracking research.
Methodology and Data Sources
The authors conducted a comprehensive scoping review of the empirical literature, focusing on 42 cross-comparative benchmarking studies. These studies provided data on how different eye-tracker models (e.g., EyeLink, Tobii, SMI) perform under various conditions, including metrics like latency, precision, and head-movement recovery.
To assess the current state of the field, the researchers analyzed a database of 207 published eye-tracking studies in the domain of judgment and decision-making. This allowed them to quantify how frequently key parameters—such as accuracy, precision, and data loss—are actually reported in peer-reviewed literature.
The Eye-Tracking Data Pipeline
Understanding the complexity of eye-tracking data is crucial for proper reporting. The process involves multiple stages, from the initial participant setup to the calculation of high-level metrics.
As shown in the figure above, data quality can be influenced at every step. Factors such as the environment (e.g., room illumination), participant characteristics (e.g., use of mascara or glasses), and the specific algorithms used for fixation and saccade detection all play a role in the final output.
Key Empirical Insights
The review highlighted several critical, yet often overlooked, factors in eye-tracking research:
The Pupil-Size Artefact
A major finding is the "pupil-size artefact." In video-based eye trackers, changes in pupil size—whether caused by ambient light or cognitive load—can be misinterpreted as eye movements. This can lead to a gaze shift of up to $5^\circ$ in the recorded data, a significant error that is rarely discussed in methodology sections.
Accuracy vs. Precision
The paper clarifies the distinction between these two fundamental metrics. Accuracy refers to how close the recorded gaze position is to the actual gaze position and is primarily influenced by the participant and the calibration process. Precision, on the other hand, refers to the reproducibility or "noise" of the signal and is largely determined by the hardware and the filters applied.
The Human Element
The skill of the eye-tracker operator is a significant variable. Experienced operators can achieve accuracy results approximately $0.2^\circ$ better than novices. Despite this impact, operator experience is almost never reported in published studies.
Results: A Call for Standardization
The analysis of the 207-study database revealed a stark disconnect between what matters for data quality and what is reported. Only 4.4% of the studies reported precision, and only 3.8% reported data loss. Furthermore, researchers were ten times more likely to report the manufacturer's specified accuracy than their own measured accuracy, even though laboratory conditions rarely match the manufacturer's "ideal" testing environment.
The Proposed "Core 9" Guideline
To address these deficiencies, the authors propose a Minimal Reporting Guideline consisting of nine essential items that should be included in every eye-tracking study:
- Sampling frequency
- Eye tracker model and manufacturer
- Calibration procedure
- Validation procedure
- Accuracy (measured in-situ)
- Precision (measured in-situ)
- Data loss and exclusion criteria
- Software and algorithms used (e.g., for fixation detection)
- Participant characteristics relevant to tracking (e.g., vision correction)
Conclusion and Future Directions
While a rigid, one-size-fits-all standard is impractical given the diversity of eye-tracking applications, these flexible guidelines provide a necessary starting point. Future work should focus on developing automated tools to help researchers calculate these metrics and moving toward mandatory standards recognized by professional bodies. Ensuring that the "empirical foundations" of eye tracking are transparently reported is a vital step toward a more robust and replicable behavioral science.
Summary updated