TL;DR
This paper introduces EYEDIAP, a novel, comprehensive database and evaluation framework designed to address the lack of standardized benchmarks for gaze estimation algorithms. By systematically varying key challenges like head pose, person identity, and environmental conditions, EYEDIAP aims to foster more robust and comparable gaze estimation research.
What Problem Does This Paper Address?
Accurate gaze estimation is crucial for a wide range of applications, from human-computer interaction and accessibility tools to understanding human behavior. However, the field has been hampered by a lack of standardized datasets for evaluating algorithms. This makes it difficult to compare different approaches, understand their strengths and weaknesses, and identify true advancements. Existing methods often rely on specialized, expensive hardware or are evaluated on disparate datasets, limiting their real-world applicability and hindering progress. This paper aims to bridge this gap by providing a publicly available, rich dataset and a common evaluation framework.
Methods
The authors introduce the EYEDIAP database, which was meticulously designed to capture data under various challenging conditions. The recording setup utilized a Kinect sensor for RGB-D data and a separate HD camera for higher resolution imagery, synchronized using LEDs. To ensure robustness, they systematically varied:
- Visual Target: Ranging from discrete and continuous targets on a screen to a 3D floating ball, mimicking different interaction scenarios.
- Head Pose: Participants were recorded with both static and mobile head poses (translation and rotation) to test robustness to head movement.
- Participant Variation: Data was collected from 16 individuals, with some sessions re-recorded under different conditions (A/B) to capture person-specific variations and environmental changes.
- Recording Conditions: Variations in ambient lighting and distance to the camera were also incorporated.
Beyond raw data, the paper details extensive data processing, including:
- RGB-D Sensor Calibration: To fuse RGB and depth information into a textured 3D surface.
- RGB-D to Screen Calibration: To map 3D camera coordinates to 2D screen coordinates.
- Head Pose and Eye Tracking: Using a 3D Morphable Model and ICP algorithm to estimate 3D head pose and eyeball locations.
- Floating Target Tracking: Providing the 3D center of the ball target for sessions using it.
- Manual Annotations: Identifying unreliable data points due to blinks or distractions.
The paper also defines a set of predefined experimental protocols to standardize evaluation, focusing on accuracy, robustness to head pose, person dependence, and condition variations.
Datasets
The primary contribution is the EYEDIAP database itself. This database is designed for both training and evaluation of gaze estimation algorithms. It is characterized by its diversity and systematic variation of challenging factors. The authors also define several benchmarks based on this database, which are intended to be used for standardized evaluation. These benchmarks allow researchers to isolate and test specific aspects of gaze estimation, such as robustness to head pose or person variability.
Key Insights and Ideas
- The critical need for standardized benchmarks: The paper strongly emphasizes that the lack of a common dataset is a major bottleneck in advancing gaze estimation research, making direct algorithm comparison nearly impossible.
- Systematic isolation of challenges: The authors' deliberate design to vary one factor at a time (head pose, person, target type, conditions) is key to understanding how each impacts gaze estimation accuracy. This allows for a more granular analysis of algorithm performance.
- RGB-D data's potential: The inclusion of depth data from RGB-D sensors opens up new avenues for more robust gaze estimation, especially in 3D environments, though challenges like low resolution remain.
- The difficulty of natural light gaze estimation: The paper highlights that even with advanced techniques, gaze estimation in natural light conditions, especially with low-resolution eye images, remains a significant challenge.
Key Results
The paper demonstrates the utility of the EYEDIAP database and protocols through an example evaluation. They compare a baseline "Head Pose" estimation (where gaze is assumed to follow head direction) against their proposed "Pose-Rectified Adaptive Linear Regression" (PR-ALR) method using RGB-D data.
- Significant improvement with PR-ALR: The PR-ALR method showed a drastic improvement in gaze estimation accuracy compared to the simple head pose assumption, as evidenced by the angular error metrics in Table 2. For instance, the average angular error for the "Head" case was around 26.1 degrees, while for PR-ALR it dropped to an average of 6.8 degrees.
- Challenging conditions: The example evaluation highlights that even with the PR-ALR method, errors can still be high, particularly in challenging scenarios like low eye image resolution (~14x10 pixels) and poor contrast (e.g., participant with dark skin).
- Data availability: Despite challenges, the recorded data provides substantial labeled samples for experimentation (e.g., around 100 seconds per recording for Protocol 1).
Limitations and Future Work
- Low Resolution Eye Images: The authors acknowledge that the eye image resolution in their dataset is quite low, which contributes to higher error rates compared to literature using high-resolution setups.
- Outlier Handling: While some manual annotations for blinks and distractions are provided, they are not exhaustive for all sessions, and outliers were not fully removed from the evaluation set in all cases.
- Future Annotations: The authors suggest that further manual annotations for unreliable data could be made if needed.
- Encouragement for Community Use: The paper strongly encourages researchers to use the EYEDIAP database and their defined protocols to foster a standardized evaluation landscape. They also invite researchers to publish their own protocols if they deviate from the provided ones.
Summary updated