Choose Your Simulator Wisely: A Review on Open-Source Simulators for Autonomous Driving

Yueyuan Li, Wei Yuan, Songan Zhang, Weihao Yan, Qiyuan Shen, Chunxiang Wang, Ming YangIEEE Trans. Intell. veh 2024 · DOI

TL;DR

This paper provides a comprehensive review of open-source simulators for autonomous driving (AD), categorizing them by task applicability and identifying critical issues in their development. It offers practical recommendations for researchers to select suitable simulators and suggests avenues for improving their fidelity and efficiency, crucial for accelerating AD development.

What Problem Does This Paper Address?

Autonomous driving (AD) systems are increasingly relying on simulators for development and testing due to the prohibitive cost and safety concerns of real-world validation. While the number of AD simulators has exploded, there's a growing concern about the validity of algorithms tested in these simulated environments. This paper addresses the need for a structured analysis of the current landscape of open-source AD simulators, aiming to bridge the gap in understanding their evolution, functionalities, limitations, and potential solutions for enhancing their credibility.

Methods

The authors conducted a thorough review of existing literature and simulators. They propose a historical categorization of simulator development into three periods: incipient, dormant, and outbreak. A taxonomy is introduced to classify simulators into five categories based on their task applicability: traffic flow, sensory data, driving policy, vehicle dynamics, and comprehensive simulators. The paper then delves into critical issues within these simulators, focusing on fidelity (sensory data, traffic scenarios, vehicle dynamics), efficiency, and data format inconsistency. For each identified issue, potential improvement methodologies are discussed, drawing from existing research and suggesting future directions.

Datasets

The paper references several real-world datasets for comparison and inspiration, including:

  • BDD100K: Used for comparing the visual fidelity of simulated RGB images against real-world data.
  • HighD, INTERACTION, Nuplan, Waymo Open Motion Dataset: These are mentioned as popular open trajectory datasets that driving policy simulators can import for realistic scenario generation.

No new datasets were introduced in this paper.

Key Insights and Ideas

  • The "Outbreak" of Simulators: The surge in open-source AD simulators since 2016 is directly linked to the rise of deep learning, which demands vast amounts of data that simulators can efficiently generate.
  • Open-Source is Key: Open-source simulators significantly lower the barrier to entry for researchers, fostering collaboration and accelerating progress in the AD field.
  • Fidelity vs. Efficiency Trade-off: There's a constant tension between achieving high-fidelity simulations (realistic sensory data, complex vehicle dynamics, nuanced traffic behaviors) and maintaining computational efficiency for large-scale training and testing.
  • Data Format Inconsistency is a Bottleneck: The lack of standardized data formats for maps and scenarios creates significant overhead for researchers, requiring manual conversion and increasing the risk of errors.
  • Critical Gaps Remain: Despite advancements, simulators still struggle with realistic adverse weather simulation, complex dynamic traffic participant behavior, and high-fidelity vehicle dynamics, impacting the reliability of sim-to-real transfer.

Key Results

The paper provides a comprehensive overview and categorization of numerous open-source AD simulators. It highlights that:

  • CARLA is identified as a leading open-source comprehensive simulator, particularly strong in sensory data simulation.
  • SUMO is the primary actively maintained open-source traffic flow simulator.
  • Gazebo is recommended as a reliable open-source option for vehicle dynamics simulation due to its robotics domain experience.
  • Table V provides a detailed comparison of actively maintained open-source driving policy simulators based on features like realistic perception, custom maps, trajectory importing, and multi-agent support.
  • Table VI compares the physics/dynamics models of simulators, noting that many rely on third-party engines or simpler bicycle models, limiting their fidelity for advanced control testing.
  • Table VII illustrates the varied support for map and scenario data formats across simulators, underscoring the inconsistency issue.
  • Table VIII details the hardware requirements and performance features (like headless simulation and distributed computing) of various simulators, indicating that many still lack optimized features for high-efficiency simulation.

Limitations and Future Work

Limitations: * Fidelity of Sensory Data: Simulators struggle to accurately replicate adverse weather conditions and the associated noise interference for all sensor types (e.g., LiDAR, radar). * Fidelity of Traffic Scenarios: Simulating realistic, dynamic, and interactive behaviors of traffic participants (vehicles, pedestrians, cyclists) remains a significant challenge. * Fidelity of Vehicle Dynamics: Open-source simulators often lack the high-fidelity vehicle dynamics models found in commercial counterparts, limiting their use for testing advanced control algorithms. * Data Format Inconsistency: The lack of standardized map and scenario data formats necessitates considerable effort in data conversion and integration. * Efficiency: Many simulators do not fully support hardware acceleration, distributed computation, or headless simulation, hindering efficient large-scale testing.

Future Work: * Standardized Interfaces: Developing standardized interfaces for sensory data simulation algorithms to simplify integration. * Improved Behavior Modeling: Enhancing traffic participant behavior modeling through techniques like Graph Neural Networks (GNNs) and adversarial methods. * Data Sharing Platforms: Establishing platforms for sharing real-world vehicle dynamics data to improve open-source simulator fidelity. * Automated Map Construction: Advancing automated map reconstruction techniques to reduce manual effort and improve efficiency. * Optimized Simulation Architectures: Focusing on software architecture and code quality to improve simulator efficiency, including better support for hardware acceleration and distributed computing.

Summary updated

BibTeX
@article{li2024choose,
  author = {Yueyuan Li and Wei Yuan and Songan Zhang and Weihao Yan and Qiyuan Shen and Chunxiang Wang and Ming Yang},
  title = {{Choose Your Simulator Wisely: A Review on Open-Source Simulators for Autonomous Driving}},
  journal = {IEEE Trans. Intell. veh},
  year = {2024},
  doi = {10.1109/tiv.2024.3374044},
}
Figure