TL;DR
This paper introduces a real-time gaze-tracking system that enhances standard RGB webcam performance by combining deep-learning-based gaze estimation with a novel "conditional residual correction" powered by body pose features. By integrating ETH-XGaze for direction and RTMPose for posture analysis, the system achieves a significant reduction in screen-coordinate error compared to existing open-source trackers, maintaining high reliability even when users shift their posture.
What Problem Does This Paper Address?
While dedicated eye-trackers using infrared sensors are highly accurate, they are expensive and cumbersome for everyday use. Standard webcams offer a ubiquitous alternative, but they face a major hurdle: posture sensitivity.
Most webcam-based systems rely on a one-time calibration where the user looks at specific points on the screen. This works well as long as the user stays perfectly still. However, in real-world scenarios, users naturally lean forward, tilt their heads, or slouch. These subtle changes in head and body position break the mathematical mapping established during calibration, leading to "gaze drift" and significant coordinate errors. The authors aim to bridge this gap by making the system "pose-aware."
Methods
The authors developed an integrated pipeline consisting of four main components:
- Gaze Estimation: The system uses XGaze-js (a lightweight version of the ETH-XGaze model) to estimate the raw horizontal and vertical gaze angles from facial landmarks detected via MediaPipe.
- Screen Mapping: To convert these angles into 2D screen coordinates, the system employs a nine-point calibration process. It uses second-order polynomial ridge regression to learn the specific relationship between a user's gaze features and their monitor setup.
- Posture Feature Extraction: Simultaneously, the system runs RTMPose asynchronously to detect key body joints. It extracts a seven-dimensional feature vector including head tilt, shoulder center position, and shoulder line inclination.
- Conditional Residual Correction: This is the core innovation. The system learns the "residual" (the error) between the baseline mapping and the actual target during calibration. During real-time use, it applies a correction offset based on the current pose. Crucially, this correction is conditional: it only activates if the detected pose features are deemed reliable and fall within the range observed during calibration, preventing erratic jumps when the pose is too extreme.
Datasets
- ETH-XGaze: Used for pre-training the base gaze estimation model. This is a large-scale dataset covering extreme head poses and gaze directions.
- User-Specific Calibration Data: Collected during the nine-point calibration phase for each session to train the ridge regression mapping and the residual correction models.
Key Insights and Ideas
- Posture as a Proxy for Error: The researchers realized that gaze error isn't random; it's often a function of the spatial relationship between the head and shoulders. By tracking the "shoulder center" and "head-to-shoulder" relative position, they can predict and subtract the drift.
- The Power of "No": One of the most important features is the system's ability to not correct. By implementing a validity check (ensuring pose data is fresh—within 150ms—and within learned bounds), the system avoids the "over-correction" trap that plagues many experimental trackers.
- Asynchronous Processing: By running the pose estimation (RTMPose) asynchronously from the gaze estimation, the system maintains real-time performance without letting the heavier pose model bottleneck the gaze frame rate.
Key Results
In a preliminary experiment with 12 target points, the proposed system was compared against three baselines: EyeGestures, gaze_tracker, and xgaze-js.
- Accuracy: The system achieved a mean screen-coordinate error of 204.24 pixels. This represents a 35.83% improvement over the baseline xgaze-js and a 57.69% improvement over gaze_tracker.
- Reliability: The system produced valid gaze estimates for 100% of the targets. In contrast, EyeGestures—while slightly more accurate on the points it did catch—only managed to produce valid output for 70.17% of the frames.
- Correction Utility: Analysis showed that residual correction was active in 82.74% of the tracking frames, proving that posture-based adjustments are frequently necessary in natural viewing conditions.
Limitations and Future Work
The primary limitation acknowledged by the authors is the small experimental scale, involving only one participant. This makes it difficult to generalize how the system handles diverse facial characteristics or more extreme movements. Additionally, the comparison systems were not tested under identical camera and calibration settings, which may affect the directness of the comparison.
Future directions include: * Expanding the study to a larger, more diverse participant pool. * Integrating 3D embedding into the RTMPose framework to better account for changes in user-to-camera distance (Z-axis movement). * Conducting ablation studies to independently quantify the contribution of the facial-feature-based vs. posture-enhanced correction models.
Summary updated