ADPerf: Investigating and Testing Performance in Autonomous Driving Systems

Tri Minh-Triet Pham, Diego Elias Costa, Weiyi Shang, Jinqiu Yang

Abstract

Autonomous Driving Systems (ADS) are safety-critical, real-time systems where the "speed of sight"—the latency of perception modules—is as vital as identification accuracy. This paper introduces ADPerf, a testing framework designed to identify performance bottlenecks by generating realistic, latency-inducing sensor data. By modeling industry-grade systems like Apollo and Autoware, we demonstrate that 3D obstacle detection is a critical bottleneck. Our findings show that even minor increases in processing time (up to 11.5ms) can trigger a 9% increase in dropped frames, causing downstream failures in trajectory prediction and potentially leading to vehicle "freezing" in complex traffic scenarios.

Introduction

In autonomous driving research, accuracy typically takes center stage: can the system correctly identify a pedestrian or a vehicle? However, for a vehicle operating in the real world, when that identification happens is just as important. Autonomous Driving Systems (ADS) are real-time pipelines. If a perception module takes too long to process a frame, the system may be forced to drop that frame to maintain synchronization with the live sensor feed.

While adversarial research often focuses on tricking classifiers, the "availability" gap—how increased computational demand degrades system reliability—remains under-explored. ADPerf addresses this by investigating how complex environments, such as crowded intersections, can overwhelm a system's capacity, leading to cascading failures in safety-critical modules.

Methodology

The researchers developed a three-phase approach to evaluate ADS performance under stress.

1. Performance Modeling

The study treated the Apollo and Autoware systems as queuing systems. Using Queuing Petri-Nets (QPN), the authors modeled the data flow from sensors through 2D and 3D detection modules to Multi-Sensor Fusion (MSF). This allowed for the simulation of various "what-if" scenarios involving different sensor frame rates and processing speeds.

2. The ADPerf Testing Framework

ADPerf generates "latency-stressing" test cases by applying realistic mutation operators to LiDAR point clouds. Unlike noise-based adversarial attacks, these operators are designed to increase computational load:

  • Adding Noise: Placing Gaussian noise near existing obstacle bounding boxes to force the detector to evaluate more candidate objects.
  • Adding Obstacles: Cloning existing obstacles to increase scene density.
  • Moving Obstacles: Shifting objects closer together to challenge the system's spatial partitioning and entity distinction.
The ADPerf framework workflow, illustrating the transition from system modeling to stress-testing 3D obstacle detection and trajectory prediction.

3. Cascading Impact Analysis

The analysis extended beyond the perception layer to measure how delayed or dropped frames affected the Trajectory Prediction module (utilizing the Trajectron++ model). This phase quantified the resulting errors in predicted paths and the overall impact on vehicle behavior.

Key Insights and Results

3D Detection as a Primary Bottleneck

Modeling confirmed that 3D obstacle detection is the slowest link in the ADS chain. While 2D detection and fusion modules are relatively efficient, 3D LiDAR processing often struggles to keep pace with the 10Hz–20Hz arrival rate of sensor data. Crucially, latency is not a flat delay; it is positively correlated with the number of detections in a scene. As environments become more crowded and the need for alertness increases, the system's performance naturally degrades.

Latency of 3D obstacle detection as a function of the number of detections, showing a clear positive correlation.

Performance Degradation and System Failure

The stress tests conducted via ADPerf yielded significant results:

  • Latency Spikes: Average 3D detection latency increased by up to 11.5ms.
  • Frame Drops: This latency spike led to a 9% rise in dropped frames.
  • Prediction Errors: In the PointPillar model, Average Displacement Error (ADE) jumped by up to 0.24m, and Final Displacement Error (FDE) by 0.34m.
  • Closed-Loop Failure: In simulations using Autoware and AWSIM, increased latency caused the ego-vehicle to "freeze" indefinitely. Because the system dropped frames, it lost the historical tracking data of surrounding vehicles, making it impossible to perform safe lane changes.

Conclusion and Future Work

ADPerf highlights a critical vulnerability in modern ADS: the trade-off between scene complexity and processing availability. A small delay in perception does not merely slow down the car; it can break the temporal continuity required for tracking and prediction.

Future work will involve expanding ADPerf to include other sensor modalities like Radar and exploring how heterogeneous hardware configurations (e.g., specialized GPUs) might mitigate these bottlenecks. Additionally, the researchers aim to develop more diverse driving scenarios to further stress-test the limits of ADS robustness beyond the constraints of current datasets like nuScenes.

Summary updated

Abstract

Obstacle detection is crucial to the operation of autonomous driving systems, which rely on multiple sensors, such as cameras and LiDARs, combined with code logic and deep learning models to detect obstacles for time-sensitive decisions. Consequently, obstacle detection latency is critical to the safety and effectiveness of autonomous driving systems. However, the latency of the obstacle detection module and its resilience to various changes in the LiDAR point cloud data are not yet fully understood. In this work, we present the first comprehensive investigation on measuring and modeling the performance of the obstacle detection modules in two industry-grade autonomous driving systems, i.e., Apollo and Autoware. Learning from this investigation, we introduce ADPerf, a tool that aims to generate realistic point cloud data test cases that can expose increased detection latency. Increasing latency decreases the availability of the detected obstacles and stresses the capabilities of subsequent modules in autonomous driving systems, i.e., the modules may be negatively impacted by the increased latency in obstacle detection. We applied ADPerf to stress-test the performance of widely used 3D obstacle detection modules in autonomous driving systems, as well as the propagation of such tests on trajectory prediction modules. Our evaluation highlights the need to conduct performance testing of obstacle detection components, especially 3D obstacle detection, as they can be a major bottleneck to increased latency of the autonomous driving system. Such an adverse outcome will also further propagate to other modules, reducing the overall reliability of autonomous driving systems.
BibTeX
@misc{pham2025adperf,
  author = {Tri Minh-Triet Pham and Diego Elias Costa and Weiyi Shang and Jinqiu Yang},
  title = {{ADPerf: Investigating and Testing Performance in Autonomous Driving Systems}},
  year = {2025},
  eprint = {2510.13078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.13078},
}

Loading related content...

Figure