ROAD: The Road Event Awareness Dataset for Autonomous Driving

Gurkirt Singh, Stephen Akrigg, Manuele Di Maio, Valentina Fontana, Reza Javanmard Alitappeh, Salman Khan, Suman Saha, Kossar Jeddi Saravi, Farzad Yousefi, Jacob Culley, Tom Nicholson, Jordan Omokeowa, Stanislao Grazioso, Andrew Bradley, Giuseppe Di Gironimo, Fabio Cuzzolin

Abstract

Humans drive in a holistic fashion which entails, in particular, understanding dynamic road events and their evolution. Injecting these capabilities in autonomous vehicles can thus take situational awareness and decision making closer to human-level performance. To this purpose, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge the first of its kind. ROAD is designed to test an autonomous vehicle's ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comprises videos originally from the Oxford RobotCar Dataset annotated with bounding boxes showing the location in the image plane of each road event. We benchmark various detection tasks, proposing as a baseline a new incremental algorithm for online road event awareness termed 3D-RetinaNet. We also report the performance on the ROAD tasks of Slowfast and YOLOv5 detectors, as well as that of the winners of the ICCV2021 ROAD challenge, which highlight the challenges faced by situation awareness in autonomous driving. ROAD is designed to allow scholars to investigate exciting tasks such as complex (road) activity detection, future event anticipation and continual learning.

Introduction

Most current autonomous driving research focuses on low-level perception—detecting bounding boxes for cars, pedestrians, or traffic lights. However, safe autonomous driving requires more than just identifying objects; it requires "situation awareness." A safe autonomous vehicle (AV) needs to interpret whether a car is braking, turning, or indicating, and precisely where that behavior is occurring relative to the road geometry.

There is currently a significant gap between low-level object detection and high-level decision-making. The ROAD dataset addresses this by providing rich, multi-label temporal annotations that allow models to understand the semantic meaning of dynamic events over time, mimicking the holistic way humans perceive the road.

The ROAD Framework

The core of the ROAD approach is the definition of a Road Event (RE) as a triplet: $E = (Ag, Ac, Loc)$.

  • Agent (Ag): The entity performing the action (e.g., Pedestrian, Car, Cyclist).
  • Action (Ac): The specific behavior being performed (e.g., Moving away, Braking, Indicating).
  • Location (Loc): The spatial context relative to the AV (e.g., In vehicle lane, On right pavement).
Conceptual diagrams of road events illustrating agents, actions, and locations.

This framework allows for a multi-label approach where a single agent can perform multiple actions simultaneously, such as a car turning right while indicating. This mirrors the complexity of real-world driving environments.

The ROAD Dataset

Built upon a subset of the Oxford RobotCar Dataset, ROAD is a large-scale resource for situational awareness:

  • Scale: 22 long-duration videos (approx. 8 minutes each), totaling 122,000 frames and 560,000 bounding boxes.
  • Richness: It features 1.7 million unique labels across 12 agent types, 23 action types, and 15 location types.
  • Diversity: The dataset covers various weather conditions (sun, rain, snow) and times of day, providing a robust testbed for model generalization.

Unlike traditional action recognition datasets like AVA or UCF-101-24, ROAD is specifically tailored to the driving domain, focusing on agents and behaviors critical for AV navigation.

Methodology: 3D-RetinaNet

To provide a baseline for the dataset, the authors propose 3D-RetinaNet, an online algorithm designed for real-time road event awareness. Unlike offline detectors that process entire video clips, 3D-RetinaNet builds "tubes" (sequences of bounding boxes) incrementally as frames arrive.

The architecture utilizes a 3D Feature Pyramid Network (3D-FPN) and can be configured with various backbones, including ResNet50, I3D (Inflated 3D), or SlowFast. The use of temporal backbones like SlowFast is crucial, as many road actions (like braking or indicating) cannot be identified from a single static frame.

Key Insights and Results

One of the most significant findings is the "Product of Marginals" hypothesis. The researchers found that training individual detectors for agents, actions, and locations and then multiplying their output scores often yielded results comparable to or better than training a single model for every possible complex triplet combination. This suggests that road event detection can be scaled efficiently without needing an exponential number of specific class labels.

Experimental results showed that:

  • Temporal Information is Vital: Models using SlowFast and I3D backbones significantly outperformed 2D-only versions, emphasizing the importance of motion cues.
  • Complexity of the Task: The baseline 3D-RetinaNet achieved a video-mAP of approximately 16-19% for full event detection. This relatively low score compared to standard object detection benchmarks highlights the inherent difficulty of holistic scene understanding.
  • Weather Robustness: While models were generally stable across weather types, "overcast" conditions typically saw the best performance due to the lack of sun glare or night-time visibility issues.

Conclusion and Future Work

ROAD provides a foundation for moving autonomous driving from simple perception toward "Theory of Mind," where vehicles can reason about the intentions of other agents. Future work will focus on expanding the dataset to more diverse urban environments and developing models for future event anticipation—predicting agent actions before they occur. Ultimately, these semantic road events are intended to serve as a bridge to more explainable and safer AV control logic.

Summary updated

Abstract

Humans drive in a holistic fashion which entails, in particular, understanding dynamic road events and their evolution. Injecting these capabilities in autonomous vehicles can thus take situational awareness and decision making closer to human-level performance. To this purpose, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge the first of its kind. ROAD is designed to test an autonomous vehicle’s ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comprises videos originally from the Oxford RobotCar Dataset, annotated with bounding boxes showing the location in the image plane of each road event. We benchmark various detection tasks, proposing as a baseline a new incremental algorithm for online road event awareness termed 3D-RetinaNet. We also report the performance on the ROAD tasks of Slowfast and YOLOv5 detectors, as well as that of the winners of the ICCV2021 ROAD challenge, which highlight the challenges faced by situation awareness in autonomous driving. ROAD is designed to allow scholars to investigate exciting tasks such as complex (road) activity detection, future event anticipation and continual learning. The dataset is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/gurkirt/road-dataset</uri> ; the baseline can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/gurkirt/3D-RetinaNet</uri> .
BibTeX
@article{singh2023road,
  author = {Gurkirt Singh and Stephen Akrigg and Manuele Di Maio and Valentina Fontana and Reza Javanmard Alitappeh and Salman Khan and Suman Saha and Kossar Jeddi Saravi and Farzad Yousefi and Jacob Culley and Tom Nicholson and Jordan Omokeowa and Stanislao Grazioso and Andrew Bradley and Giuseppe Di Gironimo and Fabio Cuzzolin},
  title = {{ROAD: The Road Event Awareness Dataset for Autonomous Driving}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3150906},
}
Figure