DROID: Driver-Centric Risk Object Identification

Chengxi Li, Stanley H. Chan, Yi-Ting Chen

TL;DR

DROID (Driver-centric Risk Object IDentification) is a new framework that identifies which objects in a traffic scene actually cause a driver to change their behavior. By combining a model of "Situation Awareness" with causal inference, the system can pinpoint risk objects—like a jaywalking pedestrian—by simulating what the driver would have done if that object weren't there.

What Problem Does This Paper Address?

Traditional autonomous driving and ADAS (Advanced Driver Assistance Systems) usually assess risk "objectively"—calculating the distance to a car or the probability of a collision. However, human driving is governed by subjective risk: the driver’s own perception of danger, which leads to specific reactions like braking or swerving.

The challenge is that identifying why a driver reacted is difficult. Existing methods often rely on expensive human annotations (like "importance" labels) or eye-tracking data. This paper addresses the gap by asking: Can we identify the "cause" of a driver's behavior change using only the "effect" (the driver's response) as a supervision signal?

Methods

The authors propose a two-stage framework rooted in the psychological model of Situation Awareness (SA), which involves three steps: Perception, Comprehension, and Projection.

  1. The Driver Behavior Model:
    • Perception: The system detects "Things" (cars, people) and "Stuff" (lanes, roads) and estimates their 3D depth.
  2. Comprehension: It uses Graph Convolutional Networks (GCNs) to model interactions between the ego-vehicle and the environment.
  3. Projection: A Temporal Recurrent Network (TRN) forecasts future interactions to predict if the driver will "Stop" or "Go."
  4. Causal Reasoning (The "DROID" Task): To identify the specific risk object, the system uses causal intervention. It systematically "removes" one object at a time from the video using partial convolutions (inpainting the background) and re-runs the behavior model. If removing an object changes the model's prediction from "Stop" back to "Go," that object is identified as the "cause" of the behavior change—the risk object.

Datasets

The researchers utilized the Honda Research Institute Driving Dataset (HDD) to create a specialized subset called HDDS.

  • HDDS: Contains over 180,000 frames for training. It includes two distinct layers: a "Goal-oriented" layer (the driver's intent, like turning left) and a "Stimulus-driven" layer (the driver's response, like stopping for a pedestrian).
  • Test Sets: They created a specific benchmark (Test2) with 630 frames where risk objects are manually annotated to evaluate the DROID task across four scenarios: Congestion, Crossing Pedestrians, Crossing Vehicles, and Parked Vehicles.

Key Insights and Ideas

  • Causality vs. Correlation: Standard deep learning models often pick up on "spurious correlations" (e.g., associating a specific intersection layout with stopping). By using causal intervention (physically removing objects), DROID ensures it identifies the actual functional cause of a behavior.
  • Intent-Awareness: The model doesn't just look at the video; it's "told" the driver's intent (e.g., "I am turning right"). This is crucial because a pedestrian on the left is a risk if you're turning left, but irrelevant if you're turning right.
  • Data Augmentation via Intervention: The authors used their "object removal" trick to synthesize new training data. By removing non-essential objects from "Go" scenarios, they forced the model to become more robust, nearly halving the model's prediction error (perplexity).

Key Results

  • Driver Response Prediction: The full model (GCN + TRN + Causal Augmentation) achieved a 95.44% mAP in predicting whether a driver would stop or go, significantly outperforming standard I3D+LSTM baselines (71.07%).
  • Risk Object Identification: In the DROID task, the causal approach was far more accurate than "attention-based" models. For example, in "Crossing Vehicle" scenarios, DROID achieved a mean accuracy (mAcc) of 32.5%, compared to just 16.8% for models that simply try to predict where a driver is looking.
  • Ablation Success: The study showed that explicitly modeling the 3D spatial relationship between objects was vital; removing the 3D constraint caused performance to drop.

Limitations and Future Work

  • Computational Complexity: Because the system has to "remove and re-test" every object in a scene, it is currently too slow for real-time use (~0.15s per object). The authors suggest moving toward "single-shot" causal inference in the future.
  • Scenario Diversity: The current benchmark focuses on vehicles and pedestrians. Future work could expand this to more complex "Stuff"-based risks, such as construction zones, traffic lights, or road debris.
  • Granularity of Response: The model currently uses a binary "Stop/Go" response. Real-world driving involves a spectrum of actions (slight braking, aggressive swerving) that could be modeled for finer risk assessment.

Summary updated

Abstract

Identification of high-risk driving situations is generally approached through collision risk estimation or accident pattern recognition. In this work, we approach the problem from the perspective of subjective risk. We operationalize subjective risk assessment by predicting driver behavior changes and identifying the cause of changes. To this end, we introduce a new task called driver-centric risk object identification (DROID), which uses egocentric video to identify object(s) influencing a driver's behavior, given only the driver's response as the supervision signal. We formulate the task as a cause-effect problem and present a novel two-stage DROID framework, taking inspiration from models of situation awareness and causal inference. A subset of data constructed from the Honda Research Institute Driving Dataset (HDD) is used to evaluate DROID. We demonstrate state-of-the-art DROID performance, even compared with strong baseline models using this dataset. Additionally, we conduct extensive ablative studies to justify our design choices. Moreover, we demonstrate the applicability of DROID for risk assessment.
BibTeX
@article{li2023droid,
  author = {Chengxi Li and Stanley H. Chan and Yi-Ting Chen},
  title = {{DROID: Driver-Centric Risk Object Identification}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=aHFjUv5YNR},
}

Loading related content...

Figure