Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis

Sourabh Vora, Akshay Rangesh, Mohan Manubhai Trivedi

Abstract

Researchers at UCSD’s LISA Lab have developed a generalized framework for estimating driver gaze zones using Convolutional Neural Networks (CNNs). By moving away from "personalized" systems that require per-driver calibration, their model achieves over 95% accuracy across different subjects and vehicle environments. Through an extensive ablative analysis, the study demonstrates that these networks can automatically learn to localize salient features like the eyes, providing a robust and efficient solution for real-time driver monitoring.

Introduction

Driver inattention is a leading cause of traffic accidents. As the industry transitions toward semi-autonomous vehicles, understanding where a driver is looking is critical for safe "handoffs" of control. Historically, gaze estimation has been hindered by the need for personalization—where systems must be calibrated for every new driver—and a reliance on brittle, hand-crafted features that fail in variable lighting. This research proposes a generalized system invariant to the driver’s physical features, the car's interior, and the camera's perspective.

Methodology and Architectures

The researchers treated gaze estimation as a classification problem, dividing the car's interior into seven discrete "zones" critical for driving safety.

The seven discrete gaze zones defined within the vehicle interior for classification.

Their approach involved a systematic ablative analysis across two dimensions:

  • Architectures: They compared four popular CNNs: AlexNet, VGG16, ResNet50, and SqueezeNet, all pre-trained on ImageNet and fine-tuned for this task.
  • Input Strategies: They tested four different ways to crop the input image: the full driver's side view (Face-embedded FoV), a tight face bounding box, the face plus surrounding context, and just the upper half of the face (eyes and forehead).

To understand the underlying mechanics of the models, the team used Class Activation Maps (CAMs) to visualize which image regions triggered specific classifications.

Datasets

The study utilized a new, large-scale Naturalistic Driving Dataset consisting of 11 long drives with 10 different subjects in two different cars, totaling over 47,000 annotated frames.

Examples from the naturalistic driving dataset showing diverse subjects and car interiors.

Crucially, they employed a "cross-subject" split—training on 7 drivers and testing on 3—to ensure the model generalized beyond specific facial features. They also validated the model on the publicly available Columbia Gaze Dataset to test zero-shot generalization in a lab-based environment.

Key Insights and Results

  • The Importance of the Eyes: The best performance (95.18% macro-average accuracy) was achieved using only the upper half of the face. This confirms that fine-grained ocular information is more critical than general head pose for distinguishing between zones like the windshield and the speedometer.
  • Small Kernels for Fine Detail: SqueezeNet and VGG16 significantly outperformed AlexNet. The authors suggest that AlexNet’s large initial convolution kernels (11x11) likely wash out subtle pixel-level changes in the pupil and eyelid.
  • Emergent Localization: CAM visualizations revealed that even when trained on a large field of view, the network automatically learned to focus on the eyes to make its decision, despite never being given explicit eye-location labels.
  • Real-Time Performance: SqueezeNet proved to be the most efficient architecture, running at 166.7 Hz on a Titan X GPU, making it highly suitable for practical automotive applications.

Conclusion and Future Work

This framework demonstrates that CNNs can overcome the traditional hurdles of personalization in gaze estimation. While the current system uses discrete zones, future work will explore continuous coordinate estimation and the use of Long Short-Term Memory (LSTM) networks to capture temporal context. Such additions could further smooth predictions and improve robustness during brief occlusions or rapid gaze transitions.

Summary updated

Abstract

Driver gaze has been shown to be an excellent surrogate for driver attention in intelligent vehicles. With the recent surge of highly autonomous vehicles, driver gaze can be useful for determining the handoff time to a human driver. While there has been significant improvement in personalized driver gaze zone estimation systems, a generalized system which is invariant to different subjects, perspectives and scales is still lacking. We take a step towards this generalized system using Convolutional Neural Networks (CNNs). We finetune 4 popular CNN architectures for this task, and provide extensive comparisons of their outputs. We additionally experiment with different input image patches, and also examine how image size affects performance. For training and testing the networks, we collect a large naturalistic driving dataset comprising of 11 long drives, driven by 10 subjects in two different cars. Our best performing model achieves an accuracy of 95.18% during cross-subject testing, outperforming current state of the art techniques for this task. Finally, we evaluate our best performing model on the publicly available Columbia Gaze Dataset comprising of images from 56 subjects with varying head pose and gaze directions. Without any training, our model successfully encodes the different gaze directions on this diverse dataset, demonstrating good generalization capabilities.
BibTeX
@article{vora2018driver,
  author = {Sourabh Vora and Akshay Rangesh and Mohan Manubhai Trivedi},
  title = {{Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis}},
  journal = {IEEE Trans. Intell. veh},
  year = {2018},
  doi = {10.1109/tiv.2018.2843120},
}
Figure