Evaluation of Video Masked Autoencoders' Performance and Uncertainty Estimations for Driver Action and Intention Recognition

Koen Vellenga, H. Joe Steinhauer, Göran Falkman, Tomas BjörklundWACV 2024

TL;DR

This paper evaluates the effectiveness of Video Masked Autoencoders (VMAEs) for recognizing driver actions and intentions from in-cabin and external video, achieving state-of-the-art results. It also investigates the ability of VMAE variants to quantify uncertainty, finding that they can indeed signal higher uncertainty for incorrect predictions, a crucial feature for safety-critical AI.

What Problem Does This Paper Address?

Traffic fatalities remain a significant global issue, and Advanced Driver Assistance Systems (ADAS) are key to reducing them by mitigating human error. A critical component of ADAS is the ability to accurately and reliably understand what a driver is doing (action recognition) and what they plan to do (intention recognition). While deep learning has shown promise, a major challenge for deploying AI in safety-critical applications like driving is ensuring the system can express when it's unsure about its predictions. This paper tackles this by evaluating VMAEs for both performance and uncertainty estimation in driver action and intention recognition tasks.

Methods

The core of this work involves using Self-Supervised Video Pre-trained (SSVP) Video Masked Autoencoders (VMAEs). These models are trained to reconstruct masked portions of video data, allowing them to learn rich, generalizable representations without explicit labels. The authors then fine-tune these pre-trained VMAEs for specific downstream tasks: driver action recognition (DAR) and driver intention recognition (DIR).

For driver intention recognition, when both in-cabin and external video streams are available, the paper employs an intermediate attention fusion mechanism. This approach learns a shared representation by allowing the model to focus on the most relevant parts of each video stream before combining them.

To address the crucial aspect of uncertainty quantification, the researchers adapt two probabilistic deep learning methods:

  • Bayes-by-Backprop (BBB) last-layer (LL): This method replaces the weights of the final layer with variational distributions to approximate the posterior distribution of the model's predictions.
  • Monte-Carlo (MC) dropout: This technique involves randomly dropping units (and their connections) during the forward pass of the network. By performing multiple forward passes with dropout enabled, the model effectively samples from an approximate posterior distribution.

Datasets

The study utilizes two established datasets:

  • Honda Research Institute Driving Dataset (HDD): This dataset provides 104 hours of naturalistic driving data, including external video footage and vehicle dynamics, labeled with eleven distinct driving actions (e.g., lane changes, turns, intersection passing).
  • Brain4Cars dataset: This dataset focuses on driver intention recognition and includes both in-vehicle and external camera views. It is labeled with five intention categories (e.g., go straight, left/right lane change, left/right turn).

Key Insights and Ideas

  • VMAEs excel at learning from raw video: The paper demonstrates that SSVP VMAEs can learn powerful representations directly from raw video without the need for pre-extracted features like optical flow, simplifying the pipeline.
  • Unified approach for DAR and DIR: The VMAE framework proves effective for both driver action and intention recognition, showcasing its versatility.
  • Attention fusion is key for multi-modal DIR: Combining in-cabin and external video streams using an attention fusion mechanism significantly boosts performance in driver intention recognition.
  • Uncertainty matters for safety: The study confirms that probabilistic VMAE variants can indeed signal higher uncertainty for incorrect predictions, a vital capability for safety-critical AI systems. This allows the system to flag potentially unreliable outputs.

Key Results

  • State-of-the-art performance: The VMAE models achieved state-of-the-art performance on both the HDD dataset for DAR (achieving a mean Average Precision (mAP) of 85.6%) and the Brain4Cars dataset for DIR (achieving an accuracy of 93.45% and an F1 score of 92.74% with attention fusion).
  • Uncertainty differentiation: For both datasets, the probabilistic VMAE models (BBB-LL and MC dropout with specific drop rates) showed higher average predictive entropy for incorrectly classified test instances compared to correctly classified ones. This indicates that the models are indeed less confident when they make mistakes.
  • Impact of input data: The study showed that while DAR performance is challenged by limited input data (suggesting the difficulty of online DAR), DIR performance generally improves with more input data, aligning with previous findings.

Limitations and Future Work

  • Computational cost of probabilistic models: While effective for uncertainty estimation, the probabilistic VMAE variants require MC sampling, which increases computational demands and may be impractical for real-time applications.
  • "Cold cognition" and interpretability: The VMAE's end-to-end approach relies on "cold cognition," meaning it learns implicitly without explicit reasoning. The authors suggest that extending the approach with explicit reasoning mechanisms would be beneficial for better interpretability and verification in safety-critical AI.
  • Comparison nuances: The paper acknowledges that direct comparisons with some prior work can be tricky due to differences in pre-training strategies, loss functions, and data augmentation.

The authors suggest that future work could explore more efficient probabilistic methods or alternative ways to achieve uncertainty estimation without the computational overhead of MC sampling. Furthermore, integrating explicit reasoning into the VMAE framework could enhance its trustworthiness for safety-critical applications.

Summary updated

BibTeX
@inproceedings{vellenga2024evaluation,
  author = {Koen Vellenga and H. Joe Steinhauer and Göran Falkman and Tomas Björklund},
  title = {{Evaluation of Video Masked Autoencoders' Performance and Uncertainty Estimations for Driver Action and Intention Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure