TL;DR
This paper presents a generalized framework for estimating driver gaze zones using Convolutional Neural Networks (CNNs), achieving state-of-the-art accuracy on a naturalistic driving dataset. By fine-tuning popular CNN architectures and analyzing various input strategies, they demonstrate a robust system that generalizes well to unseen drivers and even a completely different dataset without retraining.
What Problem Does This Paper Address?
Driver inattention is a major cause of vehicular accidents. Driver gaze is a crucial indicator of attention, and accurately estimating where a driver is looking is vital for advanced driver-assistance systems (ADAS) and autonomous vehicles, particularly for managing the handover between human and automated driving. While personalized gaze estimation systems have seen improvements, a generalized system that works across different drivers, vehicles, perspectives, and scales has been lacking. Existing methods often rely on complex pipelines and hand-crafted features that are not robust to variations in real-world driving conditions.
Methods
The core of this work involves fine-tuning pre-trained Convolutional Neural Networks (CNNs) on a custom naturalistic driving dataset. They explore four popular CNN architectures: AlexNet, VGG16, ResNet50, and SqueezeNet. The key innovation lies in their systematic "ablative analysis," where they test various combinations of:
- Input Pre-processing: They experiment with different ways to extract relevant image patches for the CNNs, including:
- Face-embedded Field of View (FoV): A large sub-image containing the driver's face and surrounding context.
- Face Detection: Using a bounding box around the detected face.
- Face + Context: Extending the face bounding box to include more surrounding information.
- Upper Half of the Face: Focusing specifically on the eye region.
- Network Fine-tuning: They fine-tune the last layers of the pre-trained CNNs to adapt them for gaze zone classification.
- Image Resolution: They investigate how increasing input image resolution impacts performance, particularly for the Face-embedded FoV approach.
A significant aspect is their demonstration that their best model can localize the driver's eyes and predict gaze zones without explicit face or eye bounding box annotations during training, effectively removing the need for a separate face detection step.
Datasets
- Custom Naturalistic Driving Dataset (NDS): This dataset was collected by the authors and comprises 11 drives from 10 subjects in two different cars. It features variations in weather, time of day, and driver demographics (age, gender). The dataset includes images from multiple cameras, but the experiments primarily use images from a camera mounted near the rearview mirror. Seven gaze zones are annotated: Forward, Right, Left, Center Stack, Rearview Mirror, Speedometer, and Eyes Closed. This dataset is notable for its naturalistic variability.
- Columbia Gaze Dataset: This publicly available dataset was used for evaluating the generalization capabilities of their best model. It contains 5,880 high-resolution images of 56 subjects with diverse ethnicities and many wearing prescription glasses. Subjects were instructed to look at specific points, creating various head pose and gaze direction configurations. Crucially, their model was evaluated on this dataset without any further training.
Key Insights and Ideas
- The Power of Fine-tuning: Fine-tuning pre-trained CNNs on a specific task like gaze estimation is highly effective, leveraging learned features from large datasets like ImageNet.
- Focusing on the Eyes is Key: The "Upper Half of the Face" input strategy consistently yielded the best results, highlighting the importance of the eye region for accurate gaze estimation. This suggests that even subtle eye movements are critical discriminators.
- Generalization Without Retraining: The model's ability to perform well on the Columbia Gaze Dataset, which has different camera setups and subjects, without any additional training is a major breakthrough. This indicates that the learned features are truly generalizable.
- CNNs Can Learn to Localize Eyes: The Class Activation Maps (CAMs) visualization reveals that the CNNs learn to focus on the driver's eyes, even when trained on broader image regions like the "Face-embedded FoV," without explicit eye annotations. This is a significant finding for simplifying the pipeline.
- SqueezeNet's Efficiency: SqueezeNet, known for its small model size, proved to be a strong performer, offering a good balance between accuracy and computational efficiency.
Key Results
- State-of-the-Art Accuracy: The best performing model, a fine-tuned SqueezeNet trained on the "Upper Half of the Face," achieved a macro-average accuracy of 95.18% on their naturalistic driving dataset, outperforming existing state-of-the-art methods.
- Cross-Subject Performance: The model demonstrated strong generalization by achieving high accuracy on subjects not seen during training.
- Columbia Gaze Dataset Performance: Without any retraining, the model successfully encoded gaze directions on the Columbia Gaze Dataset, confirming its generalization capabilities. For example, it correctly predicted "Forward" for configurations where subjects were looking slightly to the right, mirroring behaviors observed in their naturalistic data.
- No Face Detection Needed: The "Face-embedded FoV" approach, when trained on higher resolutions (625x625), achieved 92.13% accuracy while eliminating the need for a separate face detection step, enabling real-time performance.
- Inference Speed: SqueezeNet offered fast inference times (2.5ms for 224x224, 4ms for 448x448, 6ms for 625x625), making it suitable for real-time applications.
Limitations and Future Work
- Dataset Limitations: The Columbia Gaze Dataset only spans ±15° horizontally, limiting the evaluation of "Left" and "Right" gaze zones. Additionally, it lacks "Eyes Closed" examples, which were included in their naturalistic dataset.
- Future Work:
- More Gaze Zones: Expanding the number of gaze zones for finer-grained estimation.
- Temporal Context: Incorporating temporal information using Long Short-Term Memory (LSTM) networks to capture transitions between gaze zones and improve prediction robustness.
- Real-time Face Detection Integration: While their system can operate without face detection, integrating a real-time face detector could further enhance performance or be an alternative for specific applications.
Summary updated