Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems

Yao Deng, James Xi Zheng, Mengshi Zhang, Guannan Lou, Tianyi Zhang

Abstract

When developing autonomous driving systems (ADS), developers often need to replay previously collected driving recordings to check the correctness of newly introduced changes to the system. However, simply replaying the entire recording is not necessary given the high redundancy of driving scenes in a recording (e.g., keeping the same lane for 10 minutes on a highway). In this paper, we propose a novel test reduction and prioritization approach for multi-module ADS. First, our approach automatically encodes frames in a driving recording to feature vectors based on a driving scene schema. Then, the given recording is sliced into segments based on the similarity of consecutive vectors. Lengthy segments are truncated to reduce the length of a recording and redundant segments with the same vector are removed. The remaining segments are prioritized based on both the coverage and the rarity of driving scenes. We implemented this approach on an industry level, multi-module ADS called Apollo and evaluated it on three road maps in various regression settings. The results show that our approach significantly reduced the original recordings by over 34% while keeping comparable test effectiveness, identifying almost all injected faults. Furthermore, our test prioritization method achieves about 22% to 39% and 41% to 53% improvements over three baselines in terms of both the average percentage of faults detected (APFD) and TOP-K.

Introduction

Testing an Autonomous Driving System (ADS) is a data-intensive challenge. To ensure that a software update has not introduced regressions, developers typically replay hours of recorded driving data. However, a significant portion of this data is redundant—such as a vehicle traveling in a straight line on a highway for several minutes. Replaying these repetitive scenes consumes massive computational resources and time without providing additional testing value.

Existing test reduction techniques often struggle with the complexity of multi-module systems like Baidu’s Apollo, which integrate machine learning components with traditional control logic. Previous methods were often too simplistic, focusing only on road geometry, or required manual labeling of "interesting" clips, which is not scalable for industrial applications. This paper introduces STRaP (Scenario-based Test Reduction and Prioritization), a framework designed to automatically digest massive logs into a lean, high-impact test suite.

The STRaP Framework

STRaP operates through four primary stages to transform raw sensor logs into prioritized test cases:

1. Message Alignment

ADS modules, such as perception, planning, and control, often operate at different frequencies. STRaP utilizes a novel alignment algorithm to synchronize these asynchronous "publish-subscribe" communication channels into unified, time-stamped frames. This ensures that the state of the system is consistent across all modules at any given point in the recording.

2. Semantic Vectorization

Rather than analyzing raw pixels or sensor data, STRaP extracts high-level "semantic" features based on a predefined Driving Scene Schema. These features include the presence of pedestrians, traffic light states, and road signs. Each frame is converted into a compact integer vector, representing the essential characteristics of the driving scenario.

3. Test Reduction

To minimize the test suite size, STRaP employs three sub-steps:

  • Smoothing: A sliding window filters out "glitches" or momentary sensor noise that might otherwise be misinterpreted as a new scenario.
  • Slicing: The recording is cut into segments where the semantic vector remains consistent.
  • Truncating: Redundant segments are removed, and excessively long segments are truncated to a few seconds, preserving the core scenario while eliminating filler data.

4. Prioritization

STRaP calculates a "Rarity Score" for each segment. The heuristic assumes that common scenes (e.g., clear roads) are likely well-tested, whereas rare scenes (e.g., a pedestrian crossing in the rain) are more likely to trigger edge-case bugs. The framework reorders the test segments to execute these rare and feature-rich cases first, maximizing the speed of fault detection.

Experimental Evaluation

The researchers evaluated STRaP using Apollo 5.0, an industry-grade ADS, integrated with the SVL Simulator. The evaluation spanned three diverse maps: Cubetown (a simple circular road), Gomentum, and Shalun (both modeled after real-world testing grounds). To measure effectiveness, the team utilized Mutation Testing, systematically injecting 36 types of software "mutants" (bugs) into Apollo’s codebase to determine if the reduced test suite could still identify them.

Key Results

  • Efficiency Gains: STRaP reduced the length of driving recordings by 34% to 77%, depending on the map complexity and the specific module under test.
  • High Fidelity: Despite the reduction, the system maintained a 99% fault detection rate, successfully identifying nearly every bug caught by the original, full-length recordings.
  • Accelerated Discovery: The prioritization method improved the speed of fault detection (measured by APFD) by 22% to 53% compared to standard baselines, such as chronological or random ordering.

Conclusion and Future Work

STRaP demonstrates that leveraging the internal communication of multi-module ADS allows for highly effective, automated test reduction. By focusing on "rare" scenarios and eliminating redundancy, the framework significantly lowers the computational cost of regression testing without sacrificing safety.

Currently, STRaP relies on a manually defined schema to identify relevant features. Future research may explore the use of unsupervised machine learning to automatically discover significant features. Additionally, the authors plan to validate the approach on logs collected from physical vehicles on actual roads to account for higher levels of sensor noise and environmental unpredictability.

Summary updated

Abstract

When developing autonomous driving systems (ADS), developers often need to replay previously collected driving recordings to check the correctness of newly introduced changes to the system. However, simply replaying the entire recording is not necessary given the high redundancy of driving scenes in a recording (e.g., keeping the same lane for 10 minutes on a highway). In this paper, we propose a novel test reduction and prioritization approach for multi-module ADS. First, our approach automatically encodes frames in a driving recording to feature vectors based on a driving scene schema. Then, the given recording is sliced into segments based on the similarity of consecutive vectors. Lengthy segments are truncated to reduce the length of a recording and redundant segments with the same vector are removed. The remaining segments are prioritized based on both the coverage and the rarity of driving scenes. We implemented this approach on an industry level, multi-module ADS called Apollo and evaluated it on three road maps in various regression settings. The results show that our approach significantly reduced the original recordings by over 34% while keeping comparable test effectiveness, identifying almost all injected faults. Furthermore, our test prioritization method achieves about 22% to 39% and 41% to 53% improvements over three baselines in terms of both the average percentage of faults detected (APFD) and TOP-K.
BibTeX
@inproceedings{deng2022scenario,
  author = {Yao Deng and James Xi Zheng and Mengshi Zhang and Guannan Lou and Tianyi Zhang},
  title = {{Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  doi = {10.1145/3540250.3549152},
}
Figure