Abstract
A driver's gaze is critical for determining their attention, state, situational awareness, and readiness to take over control from partially automated vehicles. Estimating the gaze direction is the most obvious way to gauge a driver's state under ideal conditions when limited to using non-intrusive imaging sensors. Unfortunately, the vehicular environment introduces a variety of challenges that are usually unaccounted for - harsh illumination, nighttime conditions, and reflective eyeglasses. Relying on head pose alone under such conditions can prove to be unreliable and erroneous. In this study, we offer solutions to address these problems encountered in the real world. To solve issues with lighting, we demonstrate that using an infrared camera with suitable equalization and normalization suffices. To handle eyeglasses and their corresponding artifacts, we adopt image-to-image translation using generative adversarial networks to pre-process images prior to gaze estimation. Our proposed Gaze Preserving CycleGAN (GPCycleGAN) is trained to preserve the driver's gaze while removing potential eyeglasses from face images. GPCycleGAN is based on the well-known CycleGAN approach - with the addition of a gaze classifier and a gaze consistency loss for additional supervision. Our approach exhibits improved performance, interpretability, robustness and superior qualitative results on challenging real-world datasets.
Introduction
Estimating where a driver is looking (gaze estimation) is vital for vehicle safety systems to detect distraction or readiness to take control. While current models work well in controlled environments, real-world driving introduces significant hurdles. The authors identify two primary challenges: lighting variations (especially at night) and the presence of eyeglasses, which cause reflections and glare that obscure the eyes.
Simply training models on diverse datasets is often insufficient due to the large domain gap. Furthermore, existing eyeglass removal algorithms often prioritize aesthetic realism over functional accuracy, potentially "hallucinating" gaze directions that do not match the driver's actual focus. This paper introduces GPCycleGAN to bridge this gap by removing eyeglasses while strictly preserving gaze integrity.
Methodology
The proposed solution involves a multi-stage pipeline designed for robustness and real-time performance.
Hardware and Pre-processing
To mitigate lighting issues, the system utilizes Infrared (IR) cameras, which provide consistent imagery regardless of external light levels. However, IR can intensify reflections on glass lenses. To address this, adaptive histogram equalization is used to normalize the images before they are passed to the generative model.
Gaze Preserving CycleGAN (GPCycleGAN)
The core of the approach is the GPCycleGAN. Built upon the standard CycleGAN framework for unpaired image-to-image translation, this model adds a crucial Gaze Consistency Loss.
To ensure the gaze is preserved, the researchers use a pre-trained gaze classifier to generate Class Activation Maps (CAMs). These maps highlight the specific pixels the AI uses to determine gaze (such as the pupil and iris). The GPCycleGAN is penalized if the CAMs of the generated "cleaned" image differ significantly from the original image, forcing the generator to treat the eye region as "sacred" during the removal of frames and glare.
Classification
The final output is processed by a SqueezeNet-based classifier that maps the gaze into one of seven zones, including the forward view, mirrors, and dashboard instruments.
Experimental Results
The study utilizes the LISA Gaze Dataset, a naturalistic driving dataset featuring 13 subjects and over 336,000 frames of IR and RGB imagery across various conditions.
- Accuracy Boost: The GPCycleGAN achieved a 81.23% micro-average accuracy, a significant improvement over the 76.77% achieved by a baseline model trained without the GAN pre-processing.
- Glare Removal: Qualitatively, the model excels at removing "white-out" glare on lenses, which typically renders the eye region invisible to standard convolutional neural networks.
- Real-Time Performance: The entire pipeline—including landmark detection, GAN-based removal, and gaze classification—runs in approximately 27ms, making it suitable for real-time deployment in intelligent vehicles.
Conclusion
GPCycleGAN demonstrates that generative models can be effectively constrained to preserve critical semantic information like gaze direction. By combining IR imaging with gaze-consistent image translation, the authors provide a robust framework for driver monitoring that handles the complexities of real-world environments, offering improved safety and reliability for future automated vehicles.
Summary updated