BLVD: Building A Large-scale 5D Semantics Benchmark for Autonomous Driving

Jianru Xue, Jianwu Fang, Tao Li, Bohua Zhang, Pu Zhang, Zhen Ye, Jian Dou

TL;DR

This paper introduces BLVD, a novel large-scale benchmark for autonomous driving that goes beyond static scene understanding to capture the dynamic, interactive nature of traffic. BLVD enables research into 4D (3D + temporal) tracking, 5D (4D + interactive) event recognition, and 5D intention prediction, offering a more comprehensive evaluation of autonomous driving systems.

What Problem Does This Paper Address?

Current benchmarks for autonomous driving, while valuable, often focus on static scene understanding or limited temporal aspects. They typically excel at tasks like 2D/3D object detection and semantic segmentation but fall short in capturing the crucial dynamic evolution of surrounding objects and their interactions with the ego-vehicle. Understanding these dynamic behaviors, including how objects move over time, their intentions, and the interactive events they participate in, is paramount for developing safer and more agile autonomous vehicles. Existing datasets are often limited in scale, lack rich temporal annotations, or do not provide a unified platform for evaluating these complex dynamic tasks.

Methods

The BLVD benchmark is built upon a self-driving platform equipped with synchronized sensors, including a Velodyne HDL-64E LIDAR scanner and multi-view high-resolution cameras. The core contribution lies in the extensive and novel 5D semantic annotations. These annotations go beyond traditional 3D bounding boxes to include:

  • 4D (3D + Temporal) Tracking: Assigning unique IDs to objects across frames to track their trajectories in 3D space over time.
  • 5D (4D + Interactive) Event Recognition: Labeling specific interactive events (e.g., "overtaking from left," "pedestrian crossing slowly") that occur between surrounding objects and the ego-vehicle. These events are associated with trajectory points.
  • 5D Intention Prediction: Predicting future locations, geometrical structures, orientations, and interactive event states of surrounding objects.

The annotation process involved a semi-automatic tool and careful manual verification to ensure accuracy and consistency. The benchmark is designed to cover diverse scenarios, including urban and highway driving, under various light conditions (daytime and nighttime) and participant densities (low and high).

Datasets

The paper introduces the BLVD (Building A Large-scale 5D Semantics Benchmark) dataset. It contains:

  • 249,129 3D annotations.
  • 4,902 independent instances for tracking with a total trajectory length of 214,922 points.
  • 6,004 valid fragments for 5D interactive event recognition.
  • 4,900 individuals for 5D intention prediction.

BLVD is collected from a single city (Changshu, China) to ensure consistent data collection but covers a wide range of driving conditions. It aims to be significantly larger and more comprehensive than existing benchmarks like KITTI [2] and Cityscapes [3] in terms of dynamic scene understanding and the variety of tasks it supports.

Key Insights and Ideas

  • The Importance of 5D Semantics: The paper strongly advocates for moving beyond static 3D understanding to incorporate temporal dynamics and interactive events for truly robust autonomous driving systems.
  • Unified Platform for Dynamic Tasks: BLVD provides a single, large-scale benchmark that supports multiple interconnected dynamic tasks (tracking, event recognition, intention prediction), fostering holistic research.
  • Interactive Events as Key to Understanding: Recognizing specific interactive events is highlighted as a crucial step for enabling autonomous vehicles to make informed decisions and plan safe maneuvers.
  • Intention Prediction Beyond Location: The benchmark pushes the boundaries of intention prediction by including not just future locations but also predicting event types, geometry, and orientation, offering a more complete picture of an object's future state.

Key Results

The paper presents statistics and analyses of the BLVD dataset, demonstrating its scale and diversity. While the paper focuses on introducing the benchmark and its potential, it also provides initial experimental results for the proposed tasks:

  • 4D Tracking: The benchmark offers 654 video sequences for 4D tracking, significantly more than prior LiDAR-based tracking datasets.
  • 5D Interactive Event Recognition: The dataset includes 6,004 labeled event fragments across various scenarios. The paper shows statistics of these events across different conditions (Fig. 7 & 8).
  • 5D Intention Prediction: Initial experiments using LSTM networks for location prediction show that longer observation periods lead to more accurate predictions, while longer prediction horizons increase prediction error (Table III). The paper also defines metrics for predicting 3D bounding boxes and orientation, indicating a more challenging intention prediction task.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current benchmark focuses on participants within a 50-meter range in the front view and does not cover all possible scenarios or object types. The annotation process, while rigorous, is inherently complex and time-consuming.
  • Future Work: The authors plan to embrace the 3D detection task and integrate it into an "integrated chain" where each task can inform and improve the subsequent ones. They also aim to balance annotations for better utilization and potentially expand the dataset.

Summary updated

Abstract

In autonomous driving community, numerous benchmarks have been established to assist the tasks of 3D/2D object detection, stereo vision, semantic/instance segmentation. However, the more meaningful dynamic evolution of the surrounding objects of ego-vehicle is rarely exploited, and lacks a large-scale dataset platform. To address this, we introduce BLVD, a large-scale 5D semantics benchmark which does not concentrate on the static detection or semantic/instance segmentation tasks tackled adequately before. Instead, BLVD aims to provide a platform for the tasks of dynamic 4D (3D+temporal) tracking, 5D (4D+interactive) interactive event recognition and intention prediction. This benchmark will boost the deeper understanding of traffic scenes than ever before. We totally yield 249,129 3D annotations, 4,902 independent individuals for tracking with the length of overall 214,922 points, 6,004 valid fragments for 5D interactive event recognition, and 4,900 individuals for 5D intention prediction. These tasks are contained in four kinds of scenarios depending on the object density (low and high) and light conditions (daytime and nighttime). The benchmark can be downloaded from our project site https://github.com/VCCIV/BLVD/.
BibTeX
@inproceedings{xue2019blvd,
  author = {Jianru Xue and Jianwu Fang and Tao Li and Bohua Zhang and Pu Zhang and Zhen Ye and Jian Dou},
  title = {{BLVD: Building A Large-scale 5D Semantics Benchmark for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8793523},
}
Figure