Cyber Mobility Mirror: A Deep Learning-Based Real-World Object Perception Platform Using Roadside LiDAR

Yongkang Liu, Zhengwei Bai, Saswat Priyadarshi Nayak, Xuanpeng Zhao, Guoyuan Wu, Matthew J. Barth, Xuewei Qi, Yongkang Liu 0005, Emrah Akin Sisbot, Kentaro Oguchi

TL;DR

This paper introduces the Cyber Mobility Mirror (CMM), a novel roadside LiDAR-based system for real-time 3D object perception and reconstruction. CMM aims to enhance Cooperative Driving Automation (CDA) by providing high-fidelity environmental awareness, overcoming limitations of vehicle-only sensing.

What Problem Does This Paper Address?

Cooperative Driving Automation (CDA) promises to revolutionize transportation by improving safety and efficiency. However, current CDA systems often rely on vehicle-mounted sensors, which have limited sensing ranges and are prone to occlusions. Furthermore, the cost of equipping every vehicle with advanced sensors is prohibitive. This paper addresses these challenges by proposing an infrastructure-based perception system that leverages roadside LiDAR to create a "cyber-world" replica of the traffic environment, augmenting vehicle-centric perception.

Methods

The CMM system is comprised of six key components:

  1. Data Pre-processor: Retrieves and preprocesses raw LiDAR data.
  2. Roadside 3D Object Detector: Generates 3D detection results from LiDAR point clouds. A novel Roadside Point-cloud Encoder and Decoder (RPEaD) is introduced to bridge the gap between roadside LiDAR data and models trained on vehicle-mounted LiDAR datasets. This involves a self-calibration approach to transform point clouds into a common coordinate space.
  3. Multi-Object Tracker: Identifies and tracks detected objects. The paper proposes 3DSORT, a real-time 3D multi-object tracking method that extends DeepSORT by incorporating 3D object matching.
  4. Global Locator: Maps positioning information from the LiDAR coordinate to geographic coordinates. This involves a multi-step transformation from horizontal LiDAR coordinates to ECEF (Earth-Centered, Earth-Fixed) and then to geodetic coordinates (latitude, longitude, altitude).
  5. Cloud-Based Communicator: Transmits perception information from roadside sensors to equipped vehicles.
  6. Onboard Advisor: Reconstructs and displays real-time traffic conditions via a Graphical User Interface (GUI) on the vehicle.

Datasets

The paper mentions using the nuScenes dataset for training the detection model. For evaluation, they collected and manually labeled 130 frames of testing data from a real-world intersection, identifying 1661 vehicles as ground truth.

Key Insights and Ideas

  • Infrastructure as a Sensor Network: The core idea is to shift some of the perception burden from individual vehicles to roadside infrastructure, creating a more comprehensive and robust awareness system.
  • Bridging the Onboard-Offboard Gap: The RPEaD component is crucial for enabling the transferability of deep learning models trained on onboard LiDAR data to roadside LiDAR data, which has different spatial configurations.
  • Real-World Validation: The paper moves beyond simulation to deploy and test a functional CMM system at a real-world intersection, demonstrating its practical feasibility.
  • Geo-localization for Context: Transforming object detections into geodetic coordinates provides a universal frame of reference, essential for integrating with other systems and for visualization on maps.

Key Results

  • Detection Performance: The CMM system achieved a precision of 96.99% and a recall of 83.62% on the evaluated dataset.
  • Localization Accuracy: The average geo-localization error was 0.14m.
  • Real-time Display: Traffic conditions were displayed on the vehicle's GUI at a frequency of 3-4 Hz.
  • Latency: The total system latency was measured to be between 285ms and 335ms.

Limitations and Future Work

  • Limited Coverage: The current CMM FOS uses a single LiDAR sensor, covering only a limited area of the intersection.
  • Future Work:
    • Perception Accuracy: Improve detection accuracy by enhancing model transferability, potentially through more extensive use of simulation data for training.
  • Perception Range: Expand coverage by integrating additional sensors like cameras and deploying multiple LiDARs to cover larger areas or entire intersections and corridors.
  • Real-time Performance: Optimize software programming for tracking and localization modules and upgrade hardware for edge servers and onboard computers to further reduce latency.

Summary updated

Abstract

Object perception plays a fundamental role in Cooperative Driving Automation (CDA) which is regarded as a revolutionary promoter for the next-generation transportation systems. However, the vehicle-based perception may suffer from the limited sensing range and occlusion as well as low penetration rates in connectivity. In this paper, we propose Cyber Mobility Mirror (CMM), a next-generation real-time traffic surveillance system for 3D object perception and reconstruction, to explore the potential of roadside sensors for enabling CDA in the real world. The CMM system consists of six main components: 1) the data pre-processor to retrieve and preprocess the raw data; 2) the roadside 3D object detector to generate 3D detection results; 3) the multi-object tracker to identify detected objects; 4) the global locator to map positioning information from the LiDAR coordinate to geographic coordinate using coordinate transformation; 5) the cloud-based communicator to transmit perception information from roadside sensors to equipped vehicles, and 6) the onboard advisor to reconstruct and display the real-time traffic conditions via Graphical User Interface (GUI). In this study, a field-operational system is deployed at a real-world intersection, University Avenue and Iowa Avenue in Riverside, California to assess the feasibility and performance of our CMM system. Results from field tests demonstrate that our CMM prototype system can provide satisfactory perception performance with 96.99% precision and 83.62% recall. High-fidelity real-time traffic conditions (at the object level) can be geo-localized with an average error of 0.14m and displayed on the GUI of the equipped vehicle with a frequency of 3-4 Hz.
BibTeX
@article{liu2023cyber,
  author = {Yongkang Liu and Zhengwei Bai and Saswat Priyadarshi Nayak and Xuanpeng Zhao and Guoyuan Wu and Matthew J. Barth and Xuewei Qi and Yongkang Liu and Emrah Akin Sisbot and Kentaro Oguchi},
  title = {{Cyber Mobility Mirror: A Deep Learning-Based Real-World Object Perception Platform Using Roadside LiDAR}},
  journal = {IEEE Trans. ITS},
  year = {2023},
  doi = {10.1109/tits.2023.3268281},
}
Figure