Context-Empowered Visual Attention Prediction in Pedestrian Scenarios

Igor Vozniak, Philipp Müller, Lorena Hell, Nils Lipp, Ahmed Abouelazm, Christian MüllerWACV 2023

TL;DR

This paper introduces Context-SalNET, a novel deep learning model for predicting pedestrian visual attention in street-crossing scenarios. By explicitly incorporating context factors like urgency and safety preference, and addressing the sparsity of ground truth data with a new loss function and uncertainty modeling, Context-SalNET significantly improves upon existing saliency prediction methods.

What Problem Does This Paper Address?

Predicting where pedestrians will look is crucial for creating realistic simulations for training autonomous vehicles and human drivers, as well as for understanding traffic safety. Existing visual attention models primarily focus on "free-viewing" scenarios, where people look at static images without a specific task. These models don't account for the dynamic and context-dependent nature of pedestrian attention in real-world situations like crossing a street. Pedestrians' attention is heavily influenced by factors such as time pressure and perceived risk, and their gaze patterns are often sparse and task-specific, making them difficult to model with traditional methods. Furthermore, there's a lack of publicly available datasets specifically designed for training and evaluating pedestrian attention models in these complex scenarios.

Methods

The authors propose Context-SalNET, an encoder-decoder architecture that explicitly models context.

  • Contextual Encoding: The model augments the latent space of the encoder-decoder with information about the pedestrian's urgency and safety preference.
  • Exponentially Weighted Mean Squared Error (ew-MSE) Loss: To address the sparsity of ground truth saliency maps (where only a small portion of pixels are "hot"), a novel loss function is introduced. This loss penalizes high incorrect predictions less, preventing the model from defaulting to predicting zero everywhere.
  • Epistemic Uncertainty Modeling: Recognizing the limited availability of data for this specific task, the model incorporates epistemic uncertainty estimation using dropout during inference. This helps to quantify the model's confidence in its predictions.

Datasets

  • Newly Introduced Dataset: The paper introduces the first publicly available dataset for pedestrian visual attention in VR, specifically designed for street-crossing scenarios. This dataset includes variations in context factors (urgency and safety preference) and is derived from realistic virtual environments based on real-world accident data (GIDAS). It comprises approximately 35,000 frames.
  • SALICON: A context-free version of the model was evaluated on the SALICON dataset to compare its performance against existing saliency prediction models in a more general setting.

Key Insights and Ideas

  • Context is King for Pedestrian Attention: The study strongly emphasizes that pedestrian attention in traffic is not just about low-level visual features but is heavily driven by high-level contextual factors like urgency and safety.
  • Sparsity is a Major Challenge: Traditional saliency prediction methods struggle with the sparse nature of pedestrian gaze data, leading to poor performance. The proposed ew-MSE loss is a crucial step in overcoming this.
  • Uncertainty Matters: In data-scarce domains like this, modeling uncertainty is vital for understanding model reliability and for potential downstream applications that require confidence estimates.
  • VR as a Realistic Data Source: The use of VR and digital twins provides a controlled yet realistic environment for collecting rich pedestrian behavior data, bridging the gap between lab studies and real-world observations.

Key Results

  • State-of-the-Art Performance: Context-SalNET significantly outperforms existing state-of-the-art saliency prediction approaches on their newly introduced dataset, achieving improvements across multiple evaluation metrics (e.g., AUC-J, s-AUC, AUC-B, NSS, SIM, CC).
  • Ablation Study Success: Ablation studies confirm the effectiveness of each proposed component: context modeling, the ew-MSE loss, and epistemic uncertainty modeling all contribute to improved performance.
  • Context-Free Performance: A context-free version of Context-SalNET performs competitively on the SALICON benchmark, demonstrating the robustness of its core architecture.
  • Qualitative Improvements: Qualitative analysis shows Context-SalNET produces more accurate and relevant attention maps compared to models that rely heavily on center bias or lack contextual information.

Limitations and Future Work

  • Limitations:
    • While the VR dataset aims for realism, a domain gap may still exist between synthetic and real-world images.
  • The current model focuses on two context factors; other person-specific factors could be incorporated.
  • Future Work:
    • Extend the model to include additional context factors and different traffic participant roles (e.g., drivers, cyclists).
  • Investigate joint attention between drivers and pedestrians.
  • Explore the transferability of models trained on synthetic data to real-world scenarios.
  • Further research into the impact of different pre-trained weights and cultural differences in pedestrian behavior.

Summary updated

BibTeX
@inproceedings{vozniak2023context,
  author = {Igor Vozniak and Philipp Müller and Lorena Hell and Nils Lipp and Ahmed Abouelazm and Christian Müller},
  title = {{Context-Empowered Visual Attention Prediction in Pedestrian Scenarios}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure