Hybrid regression and isophote curvature for accurate eye center localization

Yifan Xia, Jianwen Lou, Junyu Dong, Lin Qi, Gongfa Li, Hui YuMultimedia Tools and Applications 2020 · DOI

Abstract

This paper introduces a hybrid "coarse-to-fine" framework for eye center localization that combines the robustness of regression-based machine learning with the mathematical precision of isophote curvature. By using the Supervised Descent Method (SDM) to provide a rough spatial constraint, the algorithm avoids common pitfalls like getting "distracted" by eyebrows or glasses, achieving state-of-the-art accuracy and real-time performance (45ms) on standard webcam images.

Introduction

Accurate eye center localization is a critical component for hands-free human-computer interaction, gaze estimation, and driver monitoring. While high-end infrared systems provide high accuracy, they are often expensive and perform poorly in direct sunlight. The challenge lies in achieving similar precision using standard, low-resolution webcams where factors like poor lighting, head tilt, and visual "noise" (such as hair, glasses, or shadows) make the pupil difficult to isolate.

Existing methods generally fall into two categories: appearance-based methods, which search for circular shapes or dark blobs, and model-based methods, which learn eye shapes. While appearance-based methods can be precise, they are easily fooled by eyebrows; conversely, model-based methods are robust but often lack the pixel-perfect accuracy required for fine gaze tracking. This paper proposes a hybrid approach to bridge this gap.

Methodology

The proposed system utilizes a three-stage hybrid pipeline to refine the search for the eye center from a global face image down to a specific pixel coordinate.

The proposed three-stage hybrid pipeline for eye center localization.
  1. Rough Localization (SDM): The process begins with the Supervised Descent Method (SDM). Unlike standard gradient descent, which can get trapped in local minima, SDM learns a sequence of descent directions during training. Using SIFT features, it provides a "rough" estimate of the eye region and center, acting as a robust anchor resistant to lighting changes.
  2. Candidate Generation (Isophote Curvature): Within the eye region identified in the first stage, the algorithm applies isophote curvature. Isophotes are lines of equal intensity in an image. Given that the pupil is roughly circular, the centers of these curved lines should converge at the eye center. To increase reliability, a Gaussian Pyramid processes the image at multiple scales to generate a pool of five potential candidates.
  3. Refinement (SDM-based Means of Gradient): Finally, the system combines the candidates. It employs a modified "Means of Gradient" formula that calculates the dot product of gradient vectors. Crucially, a spatial constraint based on the initial SDM estimate and a weighting factor for darkness (as pupils are typically the darkest part of the eye region) are applied to select and refine the final center point.

Experimental Results

The method was trained using the COFW (Caltech Occluded Faces in the Wild) database, which contains 1,852 images with significant variations in pose and heavy occlusion. Evaluation was conducted on two classic benchmarks: BioID (1,521 low-resolution grayscale images) and GI4E (1,236 high-quality RGB images).

Qualitative results of the proposed method across various subjects and lighting conditions.

Performance Metrics

The proposed method outperformed nearly all existing state-of-the-art (SOTA) methods on both benchmarks:

  • Accuracy: On the BioID dataset, the method achieved 88.1% accuracy for a normalized error of $e \leq 0.05$ (roughly the diameter of a pupil). At $e \leq 0.25$ (eye corner distance), it reached 100%.
  • Robustness: The hybrid approach handled reflections well; for subjects wearing glasses, accuracy remained high at 80.7% ($e \leq 0.05$ on BioID).
  • Speed: The algorithm processes an image in approximately 45ms on a standard laptop (Intel i5), making it significantly faster than many competitors and viable for real-time applications.

Discussion and Conclusion

A key insight of this research is that the SDM result serves as a vital mathematical constraint for the refinement stage, preventing high-precision gradient methods from jumping to "false positives" like dark frames of glasses. Furthermore, the use of a multi-scale Gaussian pyramid ensures that if a pupil is blurry at one resolution, it may still be detectable at another.

Despite these strengths, the method remains dependent on an initial face detection step (Viola-Jones); if the face is not detected, the localization fails. It also struggles when eyes are completely closed or under extreme reflections. Future work may involve incorporating "inter-eye" geometric constraints to further stabilize tracking and exploring deep learning architectures as more annotated data becomes available.

Summary updated

BibTeX
@article{xia2020hybrid,
  author = {Yifan Xia and Jianwen Lou and Junyu Dong and Lin Qi and Gongfa Li and Hui Yu},
  title = {{Hybrid regression and isophote curvature for accurate eye center localization}},
  journal = {Multimedia Tools and Applications},
  year = {2020},
  doi = {10.1007/s11042-019-08160-5},
}

Loading related content...

Figure