Abstract
Autonomous Driving Systems (ADS) are safety-critical, real-time systems where the "speed of sight"—the latency of perception modules—is as vital as identification accuracy. This paper introduces ADPerf, a testing framework designed to identify performance bottlenecks by generating realistic, latency-inducing sensor data. By modeling industry-grade systems like Apollo and Autoware, we demonstrate that 3D obstacle detection is a critical bottleneck. Our findings show that even minor increases in processing time (up to 11.5ms) can trigger a 9% increase in dropped frames, causing downstream failures in trajectory prediction and potentially leading to vehicle "freezing" in complex traffic scenarios.
Introduction
In autonomous driving research, accuracy typically takes center stage: can the system correctly identify a pedestrian or a vehicle? However, for a vehicle operating in the real world, when that identification happens is just as important. Autonomous Driving Systems (ADS) are real-time pipelines. If a perception module takes too long to process a frame, the system may be forced to drop that frame to maintain synchronization with the live sensor feed.
While adversarial research often focuses on tricking classifiers, the "availability" gap—how increased computational demand degrades system reliability—remains under-explored. ADPerf addresses this by investigating how complex environments, such as crowded intersections, can overwhelm a system's capacity, leading to cascading failures in safety-critical modules.
Methodology
The researchers developed a three-phase approach to evaluate ADS performance under stress.
1. Performance Modeling
The study treated the Apollo and Autoware systems as queuing systems. Using Queuing Petri-Nets (QPN), the authors modeled the data flow from sensors through 2D and 3D detection modules to Multi-Sensor Fusion (MSF). This allowed for the simulation of various "what-if" scenarios involving different sensor frame rates and processing speeds.
2. The ADPerf Testing Framework
ADPerf generates "latency-stressing" test cases by applying realistic mutation operators to LiDAR point clouds. Unlike noise-based adversarial attacks, these operators are designed to increase computational load:
- Adding Noise: Placing Gaussian noise near existing obstacle bounding boxes to force the detector to evaluate more candidate objects.
- Adding Obstacles: Cloning existing obstacles to increase scene density.
- Moving Obstacles: Shifting objects closer together to challenge the system's spatial partitioning and entity distinction.
3. Cascading Impact Analysis
The analysis extended beyond the perception layer to measure how delayed or dropped frames affected the Trajectory Prediction module (utilizing the Trajectron++ model). This phase quantified the resulting errors in predicted paths and the overall impact on vehicle behavior.
Key Insights and Results
3D Detection as a Primary Bottleneck
Modeling confirmed that 3D obstacle detection is the slowest link in the ADS chain. While 2D detection and fusion modules are relatively efficient, 3D LiDAR processing often struggles to keep pace with the 10Hz–20Hz arrival rate of sensor data. Crucially, latency is not a flat delay; it is positively correlated with the number of detections in a scene. As environments become more crowded and the need for alertness increases, the system's performance naturally degrades.
Performance Degradation and System Failure
The stress tests conducted via ADPerf yielded significant results:
- Latency Spikes: Average 3D detection latency increased by up to 11.5ms.
- Frame Drops: This latency spike led to a 9% rise in dropped frames.
- Prediction Errors: In the PointPillar model, Average Displacement Error (ADE) jumped by up to 0.24m, and Final Displacement Error (FDE) by 0.34m.
- Closed-Loop Failure: In simulations using Autoware and AWSIM, increased latency caused the ego-vehicle to "freeze" indefinitely. Because the system dropped frames, it lost the historical tracking data of surrounding vehicles, making it impossible to perform safe lane changes.
Conclusion and Future Work
ADPerf highlights a critical vulnerability in modern ADS: the trade-off between scene complexity and processing availability. A small delay in perception does not merely slow down the car; it can break the temporal continuity required for tracking and prediction.
Future work will involve expanding ADPerf to include other sensor modalities like Radar and exploring how heterogeneous hardware configurations (e.g., specialized GPUs) might mitigate these bottlenecks. Additionally, the researchers aim to develop more diverse driving scenarios to further stress-test the limits of ADS robustness beyond the constraints of current datasets like nuScenes.
Summary updated