Scenario-based test reduction and prioritization for multi-module autonomous driving systems

Yao Deng, Xi Zheng, Mengshi Zhang, Guannan Lou, Tianyi Zhang

TL;DR

Researchers have developed STRaP, a framework that slashes the time required for autonomous driving system (ADS) regression testing by over 34% (and up to 77%). By automatically identifying and prioritizing unique driving scenarios from massive sensor logs, STRaP detects 99% of software bugs in a fraction of the time compared to replaying full recordings.

What Problem Does This Paper Address?

Testing an Autonomous Driving System (ADS) is a data nightmare. To ensure a software update hasn't broken anything (regression testing), developers often replay hours of recorded driving data. However, much of this data is "boring"—think of a car driving in a straight line on a highway for 10 minutes. Replaying these redundant scenes wastes massive amounts of computational power and time.

Existing test reduction techniques often struggle with the complexity of multi-module systems like Baidu’s Apollo, which combine machine learning with traditional logic. Previous methods were either too simplistic (only looking at road shapes) or required manual labeling of "interesting" clips, which doesn't scale. The authors tackle the challenge of how to automatically "digest" these massive logs into a lean, high-impact test suite.

Methods

The authors introduce STRaP (Scenario-based Test Reduction and Prioritization), which operates in four main stages:

  1. Message Alignment: ADS modules (like perception and planning) often run at different frequencies. STRaP uses a novel algorithm to synchronize these asynchronous "publish-subscribe" communication channels into unified time-stamped frames.
  2. Semantic Vectorization: Instead of looking at raw pixels, STRaP extracts high-level "semantic" features—such as the presence of pedestrians, traffic light colors, or stop signs—based on a predefined Driving Scene Schema. Each frame is converted into a compact integer vector.
  3. Test Reduction:
    • Smoothing: It uses a sliding window to filter out "glitches" (momentary sensor errors) that would otherwise create too many tiny segments.
  4. Slicing & Truncating: It cuts the recording into segments where the scene is consistent. It then removes redundant segments and truncates lengthy ones to a few seconds.
  5. Prioritization: Not all segments are equal. STRaP calculates a "Rarity Score." It assumes that common scenes (like clear roads) are well-tested, while rare scenes (like a pedestrian crossing in the rain) are more likely to trigger bugs. It reorders the test segments to run these "rare" and "feature-rich" cases first.

Datasets

The researchers evaluated STRaP using:

  • Apollo 5.0: An industry-grade, multi-module ADS.
  • SVL Simulator: A high-fidelity simulator used to generate driving recordings.
  • Three Diverse Maps: Cubetown (simple circular road), Gomentum, and Shalun (both based on real-world autonomous testing grounds).
  • Mutation Testing: They created a benchmark by systematically injecting 36 different types of software "mutants" (bugs) into Apollo’s code to see if STRaP could still find them.

Key Insights and Ideas

  • Leveraging the "Pub-Sub" Architecture: Most modern ADSs use a middleware (like ROS or CyberRT) where modules talk to each other. STRaP realizes that the most useful information for testing isn't in the raw video, but in the messages passed between these modules.
  • The Rarity Heuristic: In software testing, "corner cases" are where bugs hide. By mathematically defining "rarity" based on how often specific feature combinations appear in a log, STRaP successfully front-loads the most "dangerous" scenarios.
  • Glitches Matter: The authors found that raw DL model outputs are noisy. Without their "smoothing" step, the system would treat every 0.1-second sensor flicker as a new "scenario," defeating the purpose of data reduction.

Key Results

  • Massive Time Savings: STRaP reduced the length of driving recordings by 34% to 77%, depending on the complexity of the map and the specific module being tested.
  • High Fidelity: Despite the massive reduction, the system maintained a 99% fault detection rate. It caught almost every bug that the original, full-length recording would have caught.
  • Faster Bug Discovery: The prioritization method improved the speed of fault detection (measured by APFD) by 22% to 53% compared to standard baselines like chronological or random ordering.

Limitations and Future Work

While highly effective, STRaP currently relies on a manually defined schema to tell it which features (like traffic lights) to look for. The authors suggest that future versions could use unsupervised machine learning to automatically discover "interesting" features without human intervention.

Additionally, while the study used high-fidelity simulators and real-world map layouts, the next step would be validating the approach on logs collected from physical vehicles on actual roads, where sensor noise and environmental unpredictability are even higher.

Summary updated

Abstract

When developing autonomous driving systems (ADS), developers often need to replay previously collected driving recordings to check the correctness of newly introduced changes to the system. However, simply replaying the entire recording is not necessary given the high redundancy of driving scenes in a recording (e.g., keeping the same lane for 10 minutes on a highway). In this pa- per, we propose a novel test reduction and prioritization approach for multi-module ADS. First, our approach automatically encodes frames in a driving recording to feature vectors based on a driving scene schema. Then, the given recording is sliced into segments based on the similarity of consecutive vectors. Lengthy segments are truncated to reduce the length of a recording and redundant segments with the same vector are removed. The remaining seg- ments are prioritized based on both the coverage and the rarity of driving scenes. We implemented this approach on an industry- level, multi-module ADS called Apollo and evaluated it on three road maps in various regression settings. The results show that our approach significantly reduced the original recordings by over 34% while keeping comparable test effectiveness, identifying almost all injected faults. Furthermore, our test prioritization method achieves about 22% to 39% and 41% to 53% improvements over three baselines in terms of both the average percentage of faults detected (APFD) and TOP-K.
BibTeX
@inproceedings{deng2022scenario,
  author = {Yao Deng and Xi Zheng and Mengshi Zhang and Guannan Lou and Tianyi Zhang},
  title = {{Scenario-based test reduction and prioritization for multi-module autonomous driving systems}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  doi = {10.1145/3540250.3549152},
}

Loading related content...

Figure