Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving

Yuqian Shao, Xiaosong Jia, Langechuan Liu, Junchi YanarXiv 2026 · arXiv

TL;DR

Bench2Drive-Speed introduces the first benchmark, dataset, and baseline models for desired-speed conditioned end-to-end autonomous driving. It demonstrates that autonomous vehicles can be trained to follow explicit user speed and overtaking commands using a novel "virtual target speed" re-annotation strategy on regular driving data, eliminating the need for expensive, privileged expert demonstrations.

What Problem Does This Paper Address?

Modern end-to-end autonomous driving (E2E-AD) models have made incredible strides, but they typically operate as a "one-size-fits-all" chauffeur. They learn a single, implicit driving style from data, meaning passengers cannot customize their ride. If a user is rushing and wants to cruise faster or overtake slow traffic, or if a cautious passenger prefers to stay safely behind a leading vehicle, current E2E-AD systems lack the interface to accommodate them.

While traditional planning-and-control pipelines can explicitly enforce speed constraints via optimization, E2E-AD systems lack a principled framework to ingest and obey explicit speed-oriented commands. Furthermore, standard autonomous driving datasets do not contain annotations for a driver's intended target speed, making it highly challenging to train and evaluate models on user controllability without compromising safety and comfort.

Fig. 1: Bench2Drive-Speed introduces target-speed commands and overtake/follow instructions, establishing the first benchmark for desired-speed conditioned autonomous driving. We quantita- tively evaluate model controllability across multiple dimensions: speed tracking, interaction strategy, comfort, traffic compliance, etc.

Methods

The authors formalize the task of Desired Speed Conditioned Autonomous Driving by introducing two high-level control inputs: a target-speed command (longitudinal velocity control) and an overtaking command (specifying whether to pass a slower vehicle or follow it).

To train and evaluate this setup, they introduce three key components:

  1. TCP-Speed Baseline: An adaptation of Trajectory-guided Control Prediction (TCP) that integrates the user's target speed and overtaking intent directly into the model's input features to guide trajectory generation.
  2. New Evaluation Metrics:
  3. Speed-Adherence Score: Measures how closely the vehicle's actual speed matches the user-specified profile using a distance-weighted average that penalizes deviations exponentially.
  4. Overtake Score: A binary success ratio evaluating whether the vehicle correctly executes an overtake or follow maneuver when triggered by a slow-moving lead vehicle.
  5. Virtual Target Speed Re-annotation: Collecting real-world expert data with explicit speed intentions is expensive and unscalable. To bypass this, the authors propose a clever workaround: they infer a "virtual target speed" from regular driving data. They extract a "tendency speed" from short-horizon future frames by identifying monotonic speed changes and then extrapolate this trend over a randomized temporal window to prevent information leakage during training.

Datasets

The paper introduces the CustomizedSpeedDataset, which consists of 2,100 complex driving routes collected in the CARLA simulator using a modified rule-based expert planner (PDM-Lite-Speed).

The dataset features:

  • A balanced distribution across difficulty levels (Medium and Hard) and overtaking choices (1,050 overtake scenes, 1,050 follow scenes).
  • 13 highly demanding scenarios adapted from the CARLA Leaderboard v2, such as dense traffic merging, junction negotiations, and obstacle circumvention.
  • Within-route command variability, meaning target speeds change dynamically across different segments of a single route to prevent the model from memorizing specific road layouts.

Key Insights and Ideas

  • Privileged Expert Data is Unnecessary: The most exciting finding is that models trained using the virtual target speed re-annotation strategy perform on par with (and sometimes better than) models trained on true, privileged expert hyperparameters. This means speed controllability can be baked into models using existing, massive real-world driving datasets.
  • Overtaking is an Interactive Bottleneck: While models easily master longitudinal speed tracking, consistently executing overtaking commands remains incredibly difficult. Overtaking requires aggressive, long-horizon interactive planning; in hard scenarios, attempting to overtake often leads to safety violations, which severely drags down the overall score.
  • Controllability Boosts Efficiency Without Sacrificing Safety: Introducing speed conditioning actually improves the vehicle's driving efficiency on standard benchmarks because the model learns to handle multiple higher target speeds, all while maintaining comparable safety and success rates to vanilla, non-conditioned models.

Key Results

The baseline models were evaluated across 48 closed-loop evaluation routes in Bench2Drive-Speed:

  • Speed Tracking: Vanilla TCP (without speed commands) achieved a Speed-Adherence Score of only 41.54. In contrast, TCP-Speed trained on expert data scored 68.79, while the model trained on the re-annotated Virtual-Short dataset achieved the highest overall score of 69.23.
  • Overtaking Performance: While the models achieved solid Overtake Scores in Medium scenarios (~56.25), performance plummeted in Hard scenarios, maxing out at 25.00 for the virtual target speed variants and a meager 6.25 for the expert-trained model.
  • Standard Benchmarks: When tested on the standard Bench2Drive-220 routes with a default 30 km/h command, TCP-Speed maintained or slightly improved its Driving Score (54.15 vs 49.30 for vanilla TCP), proving that adding user controls does not degrade core autonomous driving capabilities.

Limitations and Future Work

The primary limitation highlighted by the authors is the persistent performance gap in executing complex interactive behaviors, specifically overtaking in hazardous conditions. The current baseline models struggle to balance the aggressiveness required to pass a vehicle with the strict safety margins needed to avoid collisions.

Additionally, the monotonic trend-based virtual speed re-annotation becomes less stable when the extrapolation horizon is extended too far (Virtual-Long), introducing noise into the target speed generation. Future research will need to focus on more robust long-horizon interactive planners to make user-specified overtaking both reliable and safe.

Summary updated

Abstract

End-to-end autonomous driving (E2E-AD) has achieved remarkable progress. However, one practical and useful function has been long overlooked: users may wish to customize the desired speed of the policy or specify whether to allow the autonomous vehicle to overtake. To bridge this gap, we present Bench2Drive-Speed, a benchmark with metrics, dataset, and baselines for desired-speed conditioned autonomous driving. We introduce explicit inputs of users' desired target-speed and overtake/follow instructions to driving policy models. We design quantitative metrics, including Speed-Adherence Score and Overtake Score, to measure how faithfully policies follow user specifications, while remaining compatible with standard autonomous driving metrics. To enable training of speed-conditioned policies, one approach is to collect expert demonstrations that strictly follow speed requirements, an expensive and unscalable process in the real world. An alternative is to adapt existing regular driving data by treating the speed observed in future frames as the target speed for training. To investigate this, we construct CustomizedSpeedDataset, composed of 2,100 clips annotated with experts demonstrations, enabling systematic investigation of supervision strategies. Our experiments show that, under proper re-annotation, models trained on regular driving data perform comparably to on expert demonstrations, suggesting that speed supervision can be introduced without additional complex real-world data collection. Furthermore, we find that while target-speed following can be achieved without degrading regular driving performance, executing overtaking commands remains challenging due to the inherent difficulty of interactive behaviors. All code, datasets and baselines are available at https://github.com/Thinklab-SJTU/Bench2Drive-Speed
BibTeX
@misc{shao2026can,
  author = {Yuqian Shao and Xiaosong Jia and Langechuan Liu and Junchi Yan},
  title = {{Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving}},
  year = {2026},
  eprint = {2603.25672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.25672},
}
Figure