Results of the 2023 CommonRoad Motion Planning Competition for Autonomous Vehicles

Niklas Kochdumper, Youran Wang, Johannes Betz, Matthias AlthoffarXiv 2024 · DOI · arXiv

Abstract

The 2023 CommonRoad Motion Planning Competition provides a standardized "proving ground" for autonomous vehicle algorithms, evaluating diverse navigation strategies against hundreds of realistic traffic scenarios. This year’s results highlight a competitive performance between two distinct philosophies: a reachability-based optimization approach from Stony Brook University and a high-performance sampling-based method from the Technical University of Munich (TUM). This paper summarizes the methodologies, benchmarking environment, and key findings from the competition.

Introduction: The Benchmarking Gap in Motion Planning

In autonomous driving, motion planning acts as the "brain," determining vehicle trajectories to avoid obstacles while adhering to traffic regulations. Despite rapid progress, the field has historically lacked standardization. Researchers often evaluate algorithms using disparate hardware, vehicle models, and custom scenarios, making objective comparisons difficult.

The CommonRoad competition addresses this "benchmarking gap" by providing a level playing field. Participants must solve a uniform set of complex, interactive scenarios—ranging from urban intersections to high-speed highway merges—using identical hardware and a standardized vehicle model based on the nonlinear dynamics of a real Ford Escort.

Methodology

The competition highlighted two top-performing strategies representing different schools of thought in robotics:

1. Reachability-Based Optimization (Stony Brook University)

This approach utilizes a decision module to calculate reachable sets—mathematical envelopes representing all possible future positions of the vehicle given its physical constraints. These sets are used to identify a "driving corridor," a guaranteed safe path through traffic. Once a corridor is established, the team solves an Optimal Control Problem (OCP) to generate a smooth trajectory that respects the vehicle's complex, nonlinear physics.

2. Sampling-Based Planning (TU Munich - FRENETIX)

The FRENETIX planner employs a high-throughput sampling approach, generating thousands of candidate trajectories simultaneously. To enhance computational efficiency, these samples are initially calculated in curvilinear coordinates relative to the road's geometry. The system then subjects these candidates to a rigorous filtering process, checking for physical feasibility, collision avoidance, and passenger comfort. The optimal surviving path is selected for execution.

Datasets and Evaluation Environment

The competition leverages the CommonRoad benchmark suite, which includes over 500 scenarios for training and testing. The Phase II evaluation set consisted of 230 interactive scenarios previously unknown to the participants.

A sequence of snapshots (t=0s to t=6s) from a CommonRoad interactive scenario showing an ego vehicle navigating a complex intersection.

To ensure realism, the SUMO (Simulation of Urban MObility) simulator is used to model traffic. Unlike static datasets, other traffic participants (cars, buses, bicycles) react dynamically to the ego vehicle’s behavior, creating a truly interactive environment.

Key Insights and Results

The 2023 competition revealed several critical insights into the current state of motion planning:

  • Convergence of Performance: Despite fundamentally different mathematical frameworks—optimization vs. sampling—both top teams achieved remarkably similar results. This suggests that the field is maturing, with different methodologies now capable of solving the same high-level complexities.
  • Computational Efficiency: The TUM team demonstrated the power of optimized implementation. Their C++ sampling-based planner processed 90,000 trajectories in just 717ms, nearly 15 times faster than a comparable Python-based version.
  • Multi-Objective Optimization: Success in the competition requires balancing multiple factors. The cost function penalizes Jerk (for comfort), Steering Rate, Lane Center Offset, and insufficient Distance to Obstacles.

Final Standings

The race for the top spot was exceptionally close:

  • Winner: Technical University of Munich (FRENETIX) solved 130 scenarios and achieved the lowest cost solution in 84 of them.
  • Runner-up: Stony Brook University solved 116 scenarios, securing the top solution in 54.

Data analysis showed that certain scenarios favored specific algorithms. For example, Stony Brook’s optimization approach excelled in specific urban environments in Frankfurt, while TUM’s sampling approach performed better in others. This suggests that future "hybrid" planners, combining both philosophies, may offer the most robust performance.

Limitations and Future Directions

While the competition provides a rigorous benchmark, several areas for future development remain:

  • Industry Participation: Broader involvement from industrial players would further validate these academic benchmarks.
  • Compute Budgets: Current planners operated within a 6-hour window for 230 scenarios. Real-world deployment requires even tighter, millisecond-level latency.
  • Traffic Law Compliance: Future iterations could incorporate more sophisticated traffic law modeling, such as complex right-of-way rules at unsignaled intersections, beyond simple cost-function penalties.

Summary updated

Abstract

In recent years, different approaches for motion planning of autonomous vehicles have been proposed that can handle complex traffic situations. However, these approaches are rarely compared on the same set of benchmarks. To address this issue, we present the results of a large-scale motion planning competition for autonomous vehicles based on the CommonRoad benchmark suite. The benchmark scenarios contain highway and urban environments featuring various types of traffic participants, such as passengers, cars, buses, etc. The solutions are evaluated considering efficiency, safety, comfort, and compliance with a selection of traffic rules. This report summarizes the main results of the competition.
BibTeX
@misc{kochdumper2024results,
  author = {Niklas Kochdumper and Youran Wang and Johannes Betz and Matthias Althoff},
  title = {{Results of the 2023 CommonRoad Motion Planning Competition for Autonomous Vehicles}},
  year = {2024},
  eprint = {2411.06425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.06425},
}
Figure