Remote Gaze Tracking System on a Large Display

Hyeon Lee, Won Lee, Chul Koo Cho, Su Gwon, Kang Park, Hee-Kyung Lee, Jihun ChaSensors 2013 · DOI

TL;DR

This paper introduces a novel remote gaze tracking system designed for intelligent TVs, overcoming limitations of previous systems by using a dual-camera setup with panning and tilting capabilities. The system achieves accurate gaze tracking (±0.737° to ±0.775°) at a distance of 1.8-2.2 meters with a processing speed of 5-10 frames/s, enabling more natural user interaction with large displays.

What Problem Does This Paper Address?

The increasing popularity of smart TVs and large displays has led to a demand for more intuitive interfaces than traditional remote controls. Gaze tracking offers a promising solution, but existing remote gaze tracking systems often struggle with large display sizes and user head movement. Many systems require users to be at a fixed, close distance, or they have limited capture volumes, restricting natural viewing positions. Furthermore, complex calibration procedures and the inconvenience of wearable devices have hindered widespread adoption. This paper aims to address these issues by developing a remote gaze tracking system that offers a larger capture volume, greater freedom of head movement, and simpler calibration for intelligent TV interfaces.

Methods

The core innovation of this system lies in its dual-camera approach and integrated mechanical movement:

  • Dual-Camera System: A Wide View Camera (WVC) captures the user's face and eye position, and importantly, estimates the Z-distance (depth) based on facial width. A Narrow View Camera (NVC) then captures a magnified, auto-focused image of the eye.
  • Panning and Tilting Mechanisms: Both cameras are mounted on a platform that can pan and tilt. This allows the system to follow the user's gaze across a larger area of the display and accommodate head movement without requiring recalibration.
  • Parallel Structure: The WVC and NVC are arranged in a parallel structure. This design simplifies calibration between the two cameras and enhances the capture speed of the NVC.
  • Auto-Focusing: The NVC employs an auto-focusing mechanism that leverages both the estimated Z-distance from the WVC and a calculated focus score from the NVC's eye image. This ensures a clear image of the eye for accurate pupil and specular reflection detection.
  • Gaze Detection Algorithm: The system uses an AdaBoost algorithm for initial face and eye detection in the WVC image. For pupil and specular reflection detection in the NVC image, a rapid eye detection (RED) method is employed, followed by circular edge detection (CED).
  • Geometric Transformation: A geometric transformation model is used to map the detected eye gaze point to the corresponding location on the TV screen, accounting for parameters like kappa (the angle between the pupillary and visual axes).

Datasets

The paper does not explicitly mention training datasets. Instead, it focuses on experimental evaluation using data collected from human subjects. The experiments involved:

  • User-dependent calibration: Subjects performed calibration by looking at specific points on the TV screen.
  • Gaze tracking accuracy measurement: 10 subjects participated, each looking at 84 reference points on a 60-inch display multiple times.
  • Head movement analysis: Subjects were equipped with Polhemus position tracking sensors to measure head movement velocity under various sitting positions.

The paper introduces a benchmark for evaluating gaze tracking accuracy on a large display with natural head movement, using 84 reference points for more intensive measurement compared to previous studies that often used only nine.

Key Insights and Ideas

  • Dual-camera synergy for large displays: Combining a wide-angle camera for initial detection and distance estimation with a narrow-angle, auto-focusing camera for detailed eye imaging is crucial for robust gaze tracking at a distance.
  • Mechanical movement is key for freedom: Panning and tilting mechanisms are essential for accommodating natural head movements and expanding the usable range of the gaze tracking system, making it more practical for TV viewing.
  • Simplified calibration through parallel structure: Aligning the cameras in parallel not only reduces complexity but also speeds up the overall process, which is vital for real-time applications like TV interfaces.
  • Auto-focusing based on multiple cues: Relying on both facial width (from WVC) and image focus score (from NVC) for auto-focusing provides a more robust solution, especially given the varying distances and potential for slight head movements.
  • Accuracy is sufficient for practical use: The achieved accuracy of around ±0.7 degrees is demonstrated to be sufficient for selecting from a large number of menu items on a 60-inch screen, indicating practical usability.

Key Results

  • Gaze Tracking Accuracy: The system achieved an average gaze tracking error of ±0.737° to ±0.775° at a distance of 1.8-2.2 meters. This is reported to be smaller than errors found in previous studies.
  • Processing Speed: The system operates at a speed of 5-10 frames/s, which is suitable for real-time interaction.
  • Head Movement Tolerance: The system allows for a wider range of head movement freedom compared to previous systems, with measured maximum head movement velocities of up to 2.887 cm/s (X-axis), 3.142 cm/s (Y-axis), and 1.305 cm/s (Z-axis) without significant degradation in performance.
  • Panning and Tilting Accuracy: Experimental results showed that pupil centers were successfully kept within the NVC image after panning and tilting, indicating the effectiveness of these mechanisms.

Limitations and Future Work

  • User-Dependent Calibration: While the system aims for simpler calibration, it still requires a user-dependent calibration stage. Errors in this stage (e.g., not gazing at the exact center) can lead to increased gaze tracking errors, as observed in some users.
  • Z-distance Estimation Accuracy: The accuracy of Z-distance estimation from facial width in the WVC image can degrade at larger distances due to reduced image resolution.
  • Future Work: The authors suggest researching methods for more accurate Z-distance measurement, potentially using additional devices like NIR-based distance measuring devices or high-cost laser scanners. They also aim to further improve the gaze tracking accuracy.

Summary updated

Abstract

We propose a new remote gaze tracking system as an intelligent TV interface. Our research is novel in the following three ways: first, because a user can sit at various positions in front of a large display, the capture volume of the gaze tracking system should be greater, so the proposed system includes two cameras which can be moved simultaneously by panning and tilting mechanisms, a wide view camera (WVC) for detecting eye position and an auto-focusing narrow view camera (NVC) for capturing enlarged eye images. Second, in order to remove the complicated calibration between the WVC and NVC and to enhance the capture speed of the NVC, these two cameras are combined in a parallel structure. Third, the auto-focusing of the NVC is achieved on the basis of both the user's facial width in the WVC image and a focus score calculated on the eye image of the NVC. Experimental results showed that the proposed system can be operated with a gaze tracking accuracy of ±0.737°~±0.775° and a speed of 5~10 frames/s.
BibTeX
@inproceedings{lee2013remote,
  author = {Hyeon Lee and Won Lee and Chul Koo Cho and Su Gwon and Kang Park and Hee-Kyung Lee and Jihun Cha},
  title = {{Remote Gaze Tracking System on a Large Display}},
  booktitle = {Sensors},
  year = {2013},
  doi = {10.3390/s131013439},
}
Figure