Estimating fusion weights of a multi-camera eye tracking system by leveraging user calibration data

Nuri Murat Arar, Jean-Philippe ThiranETRA 2016 · DOI

TL;DR

This paper tackles the accuracy limitations of simple cross-ratio (CR)-based eye tracking by proposing a novel multi-camera system. The key innovation is a weighted fusion method that learns optimal camera weights from user calibration data, significantly improving gaze estimation accuracy with minimal calibration effort.

What Problem Does This Paper Address?

Traditional eye tracking methods fall into two main categories: interpolation-based and model-based. Interpolation methods are simpler but less accurate, especially with head movement. Model-based methods are more accurate but require complex setups and calibration. Cross-ratio (CR)-based methods offer a compromise, allowing free head motion without complex camera calibration, but often suffer from lower accuracy and robustness due to simplifying assumptions. While multi-camera systems have been explored, their potential for enhancing CR-based methods through intelligent fusion of data from multiple viewpoints remains underexplored. This paper aims to bridge this gap by developing a multi-camera system that leverages user-specific calibration data to intelligently fuse information from individual cameras, thereby boosting accuracy and maintaining user-friendliness.

Methods

The proposed system utilizes a multi-camera setup (three cameras in this case) where each camera independently performs CR-based gaze estimation. The core novelty lies in the weighted fusion method for combining the gaze points (Points of Regard or PoRs) from these individual cameras. Instead of simple averaging or arbitrary weighting, this method learns optimal fusion weights by analyzing user calibration data.

Specifically, the authors extract statistical indicators from the calibration process, such as:

  • Calibration accuracy per sensor: How close the calibrated gaze points are to the actual target points for each camera.
  • Gaze availability: The reliability of feature detection for each sensor.
  • Histogram of best-performing sensors: How often each camera provides the most accurate result for a given calibration point.

These indicators are used to generate weight maps for each camera and eye. During testing, the PoRs from individual cameras are combined using a weighted average, where the weights are derived from these learned maps. This approach is independent of the underlying gaze estimation algorithm, meaning CR can be replaced with other methods. For bias correction within each camera's estimation, a linear regression-based method is employed, which is noted for its efficiency and generalization capabilities.

Datasets

The experiments were conducted using data collected from ten human users. For each user, calibration data and test data were acquired separately.

  • Calibration Data: Users looked at 25 uniformly distributed target points on a screen.
  • Test Data: Users looked at 18 target points arranged in a 3x3 grid covering the screen.

The paper does not introduce a new public benchmark dataset but rather uses data collected from participants in a controlled experimental setting.

Key Insights and Ideas

  • Calibration Data is Gold: The most significant insight is that user calibration data, typically collected to correct individual camera biases, can be effectively repurposed to learn optimal fusion weights for a multi-camera system. This avoids an additional calibration step for fusion.
  • Intelligent Fusion Outperforms Simple Averaging: Weighted fusion based on calibration statistics significantly outperforms simple averaging of gaze points from multiple cameras.
  • Subject-Specific Weighting is Slightly Better: While subject-independent weighting also improves performance, subject-specific weighting yields slightly better results, suggesting that individual user characteristics play a role, though hardware factors (camera positions, viewing angles) seem to dominate.
  • Minimal Calibration for Good Results: The system achieves competitive accuracy with as few as 5 calibration points, making it user-friendly and practical.

Key Results

  • Accuracy Improvement: The proposed multi-camera system with weighted fusion achieved an average gaze estimation accuracy error of 0.82° with only 5 calibration points. This is competitive with more complex 3D model-based systems.
  • Comparison to Baselines: The weighted fusion method significantly improved accuracy compared to non-weighted fusion and single-camera systems.
  • Real-time Performance: The system can run at approximately 30 fps on a standard PC without speed optimization, making it suitable for real-time applications.
  • Gaze Availability: The multi-camera setup inherently increases gaze availability, as a user must be outside the field of view of all cameras to lose tracking, compared to a single camera system.

Limitations and Future Work

  • Head/Body Movement Robustness: While the system allows for free head movement, the current user experiments did not involve large body movements. The authors acknowledge that further studies are needed to assess robustness against more significant head and body motion.
  • Future Work: The authors plan to investigate the robustness of their method against larger head/body movements in future work.

Summary updated

BibTeX
@inproceedings{arar2016estimating,
  author = {Nuri Murat Arar and Jean-Philippe Thiran},
  title = {{Estimating fusion weights of a multi-camera eye tracking system by leveraging user calibration data}},
  booktitle = {ETRA},
  year = {2016},
  doi = {10.1145/2857491.2857510},
}
Figure