MAAD: A Model and Dataset for "Attended Awareness" in Driving

Deepak Gopinath, Guy Rosman, Simon Stent, Katsuya Terahata, Luke Fletcher, Brenna Argall, John J. Leonard

Abstract

We propose a computational model to estimate a person's attended awareness of their environment. We define attended awareness to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely to be physically aware of. Our model takes as input scene information in the form of a video and noisy gaze estimates, and outputs visual saliency, a refined gaze estimate, and an estimate of the person's attended awareness. In order to test our model, we capture a new dataset with a high-precision gaze tracker including 24.5 hours of gaze sequences from 23 subjects attending to videos of driving scenes. The dataset also contains third-party annotations of the subjects' attended awareness based on observations of their scan path. Our results show that our model is able to reasonably estimate attended awareness in a controlled setting, and in the future could potentially be extended to real egocentric driving data to help enable more effective ahead-of-time warnings in safety systems and thereby augment driver performance. We also demonstrate our model's effectiveness on the tasks of saliency, gaze calibration, and denoising, using both our dataset and an existing saliency dataset.

Introduction

Most current driver monitoring systems focus on a "snapshot" of attention: where is the driver looking right now? However, human awareness is inherently temporal. Drivers do not forget a car in their peripheral vision the instant they look away; they maintain a "cognitive buffer" of their surroundings.

The challenge is that this internal state is invisible. While standard saliency models predict what might be important, they do not capture what a specific driver has actually perceived and retained. There is a significant gap in modeling how gaze history, scene dynamics, and cognitive load (such as distraction) combine to form a driver's actual awareness.

Methodology: Modeling Attended Awareness

The proposed MAAD model is a 3D Convolutional Neural Network (CNN) architecture based on an encoder-decoder structure. The approach is unique in its adherence to axiomatic considerations of human cognition: awareness follows gaze, awareness decays over time, awareness moves with objects (object permanence), and humans have a limited cognitive capacity.

High-level overview of the MAAD model, illustrating the integration of scene information and noisy gaze to produce saliency, refined gaze, and awareness estimates.

Architecture and Axiomatic Design

The model treats gaze not as the "ground truth" of awareness, but as a noisy signal filtered through the context of the scene. By injecting optical flow into the decoder, the model achieves object permanence: if a driver looks at a pedestrian and then looks away, the awareness heatmap "sticks" to the pedestrian as they move, rather than staying static on the pixels where the gaze occurred.

The MAAD architecture utilizes an S3D encoder-decoder framework with side-channel features for gaze and optical flow to maintain object permanence and refine gaze estimates.

Furthermore, the model exhibits a "snap-to-saliency" effect. If a sensor incorrectly reports a driver looking at empty asphalt when a salient vehicle is nearby, the model's internal saliency understanding can "snap" the gaze estimate to the likely intended target, effectively denoising the input.

The MAAD Dataset

To support this research, the authors introduced the MAAD Dataset, a large-scale resource for the community:

  • Scale: 24.5 hours of high-precision gaze data from 23 subjects.
  • Source: Road-facing videos from the DR(eye)VE dataset, augmented with ultra-precise lab-recorded tracks using an EyeLink 1000 Plus.
  • Modifiers: Subjects watched videos under various conditions, including blurred frames, flipped views, and "reading-text" distraction tasks to simulate different mental states.
  • Annotations: Includes ~54,000 third-party annotations where humans estimated a subject's awareness level based on their observed scan path.

Key Results

  • Awareness Prediction: The MAAD model significantly outperformed baseline methods. In predicting annotated awareness levels, it achieved a Mean Squared Error (MSE) of 0.138, compared to 0.425 for a standard filtered-gaze approach.
  • Gaze Denoising: The model is remarkably robust to noise. Even when input gaze was heavily corrupted, MAAD recovered the true gaze point with an error of only 22.1 pixels, compared to 51.7 pixels for the raw noisy input.
  • Saliency Benchmarking: Despite being a multi-task model, it remains competitive on pure saliency tasks, comparing favorably to state-of-the-art models on the DR(eye)VE benchmark.

Conclusion and Future Work

While the current model was trained on passive lab data, it demonstrates the potential for estimating internal cognitive states from external signals. Future work aims to extend this to real egocentric driving data where the subject is in active control of the vehicle. Integrating these awareness estimates into Active Safety Systems could provide "ahead-of-time" warnings when a driver is likely unaware of a developing hazard, significantly augmenting driver performance and safety.

Summary updated

Abstract

We propose a computational model to estimate a person's attended awareness of their environment. We define attended awareness to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely to be physically aware of. Our model takes as input scene information in the form of a video and noisy gaze estimates, and outputs visual saliency, a refined gaze estimate, and an estimate of the person's attended awareness. In order to test our model, we capture a new dataset with a high-precision gaze tracker including 24.5 hours of gaze sequences from 23 subjects attending to videos of driving scenes. The dataset also contains third-party annotations of the subjects' attended awareness based on observations of their scan path. Our results show that our model is able to reasonably estimate attended awareness in a controlled setting, and in the future could potentially be extended to real egocentric driving data to help enable more effective ahead-of-time warnings in safety systems and thereby augment driver performance. We also demonstrate our model's effectiveness on the tasks of saliency, gaze calibration, and denoising, using both our dataset and an existing saliency dataset. We make our model and dataset available at https://github.com/ToyotaResearchInstitute/att-aware/.
BibTeX
@inproceedings{gopinath2021maad,
  author = {Deepak Gopinath and Guy Rosman and Simon Stent and Katsuya Terahata and Luke Fletcher and Brenna Argall and John J. Leonard},
  title = {{MAAD: A Model and Dataset for "Attended Awareness" in Driving}},
  booktitle = {ICCVW},
  year = {2021},
  url = {https://openreview.net/forum?id=oczxiKXBeq},
}
Figure