Non-Intrusive Gaze Tracking Using Artificial Neural Networks

Shumeet Baluja, Dean PomerleauNeurIPS 1993

This classic 1994 paper from Carnegie Mellon University presents a pioneering approach to gaze tracking that prioritizes user comfort without sacrificing significant precision. By leveraging Artificial Neural Networks (ANNs), the authors moved away from the clunky hardware requirements of the era—like chin rests and head-mounted cameras—to create a system that tracks where a user is looking using nothing but a standard camera and clever machine learning.

Executive Summary

The researchers developed a non-intrusive gaze tracking system using a three-layer neural network that maps eye images directly to screen coordinates. By including information from the surrounding eye socket and using a "divided" hidden layer architecture, they achieved an accuracy of 1.5 degrees while allowing the user to move their head freely.

The Challenge: Escaping the Chin Rest

In the early 90s, if you wanted to track someone's gaze, you usually had to treat them like a laboratory specimen. "Intrusive" systems required users to wear heavy headgear, use special contact lenses, or stay perfectly still in a chin rest. These constraints were necessary because traditional computer vision algorithms relied on a fixed geometric relationship between the camera and the eye to calculate the angle of the pupil.

The authors sought to break these chains. They wanted a "non-intrusive" system that could handle the noisy and infinitely large set of images produced when a user is allowed to move their head naturally. The challenge was creating a model flexible enough to map the changing appearance of an eye to a specific (X, Y) coordinate on a monitor.

System Architecture and Methodology

The system follows a pipeline that balances traditional image processing with neural learning:

  1. Eye Localization: The system first finds the "specular reflection" (the glint of a stationary light source) on the cornea to locate the eye within the larger face image.
Specular reflection used to locate the eye
  1. Feature Extraction: A small window (either 15x15 or 15x40 pixels) is cropped around the eye.
  2. The Neural Network: These raw pixels are fed into a 3-layer feed-forward ANN.
  3. Input: Grayscale pixel values.
  4. Hidden Layer: The authors experimented with "divided" hidden layers, where one set of units predicts the X-coordinate and another predicts the Y-coordinate. This reduced the number of connections and boosted processing speed.
  5. Output: They used a Gaussian output representation (borrowed from the ALVINN autonomous driving project). Instead of two numbers, the network outputs two vectors of 50 units each. The "peak" of activation in these vectors represents the X and Y coordinates.
Neural network input and Gaussian output representation
  1. Refinement: After training, they implemented an "offset table"—a calibration step where the user tracks a cursor to correct for systematic errors in the network's predictions.

Data Collection and Training

The researchers collected their own data for this study to ensure robust training and testing:

  • Training Set: 2,000 image/position pairs collected while a user followed a cursor moving in a horizontal zig-zag pattern.
  • Testing Set: 2,000 pairs collected during a vertical zig-zag pattern to ensure the model wasn't just memorizing the training path.
  • Multi-Position Sets: To improve generalization, they later collected data from different head positions and mixed them, proving that seeing the eye from multiple angles helps the network handle head mobility.

Key Insights and Innovations

  • Context Matters (The Eye Socket): One of the biggest breakthroughs was realizing that 15x15 images of just the pupil and cornea weren't enough. Expanding the input to 15x40 to include the eye socket significantly improved accuracy. The network used the position of the eye relative to the socket as a crucial cue for gaze direction.
  • Architectural Efficiency: The researchers found that a "divided" hidden layer (splitting X and Y tasks) performed as well as or better than a fully connected layer. This suggests that the features needed to determine horizontal gaze are somewhat independent of those needed for vertical gaze.
  • Generalization through Diversity: Training on a single head position led to high errors if the user moved. By "mixing" training data from multiple sessions with different head positions, the network's ability to generalize improved drastically.

Performance and Practical Application

  • Accuracy: The system achieved an on-line accuracy of 1.5 degrees. While the best intrusive systems of the time hit 0.75 degrees, this was a remarkable result for a system that allowed head movement.
  • Speed: The system operated at 15 Hz (15 frames per second) on a Sun SPARC 10 machine. In 1994, this was fast enough for real-time interaction.
  • Practicality: The authors successfully used the system as a mouse replacement for selecting windows in an X-Window environment and for Human-Computer Interaction (HCI) studies.

Limitations and Future Directions

The authors noted that the system was still highly sensitive to significant camera movement. If the camera itself was moved between training and testing, the error spiked.

While they made strides in "head-position independence," the system still required a per-user calibration (the 2,000-image training session). The authors pointed toward future work aimed at making the system truly "user-independent," where a pre-trained network could work for a new person with little to no calibration.

Summary updated

BibTeX
@inproceedings{baluja1993non,
  author = {Shumeet Baluja and Dean Pomerleau},
  title = {{Non-Intrusive Gaze Tracking Using Artificial Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure