TL;DR
Researchers have developed STRaP, a framework that slashes the time required for autonomous driving system (ADS) regression testing by over 34% (and up to 77%). By automatically identifying and prioritizing unique driving scenarios from massive sensor logs, STRaP detects 99% of software bugs in a fraction of the time compared to replaying full recordings.
What Problem Does This Paper Address?
Testing an Autonomous Driving System (ADS) is a data nightmare. To ensure a software update hasn't broken anything (regression testing), developers often replay hours of recorded driving data. However, much of this data is "boring"—think of a car driving in a straight line on a highway for 10 minutes. Replaying these redundant scenes wastes massive amounts of computational power and time.
Existing test reduction techniques often struggle with the complexity of multi-module systems like Baidu’s Apollo, which combine machine learning with traditional logic. Previous methods were either too simplistic (only looking at road shapes) or required manual labeling of "interesting" clips, which doesn't scale. The authors tackle the challenge of how to automatically "digest" these massive logs into a lean, high-impact test suite.
Methods
The authors introduce STRaP (Scenario-based Test Reduction and Prioritization), which operates in four main stages:
- Message Alignment: ADS modules (like perception and planning) often run at different frequencies. STRaP uses a novel algorithm to synchronize these asynchronous "publish-subscribe" communication channels into unified time-stamped frames.
- Semantic Vectorization: Instead of looking at raw pixels, STRaP extracts high-level "semantic" features—such as the presence of pedestrians, traffic light colors, or stop signs—based on a predefined Driving Scene Schema. Each frame is converted into a compact integer vector.
- Test Reduction:
- Smoothing: It uses a sliding window to filter out "glitches" (momentary sensor errors) that would otherwise create too many tiny segments.
- Slicing & Truncating: It cuts the recording into segments where the scene is consistent. It then removes redundant segments and truncates lengthy ones to a few seconds.
- Prioritization: Not all segments are equal. STRaP calculates a "Rarity Score." It assumes that common scenes (like clear roads) are well-tested, while rare scenes (like a pedestrian crossing in the rain) are more likely to trigger bugs. It reorders the test segments to run these "rare" and "feature-rich" cases first.
Datasets
The researchers evaluated STRaP using:
- Apollo 5.0: An industry-grade, multi-module ADS.
- SVL Simulator: A high-fidelity simulator used to generate driving recordings.
- Three Diverse Maps: Cubetown (simple circular road), Gomentum, and Shalun (both based on real-world autonomous testing grounds).
- Mutation Testing: They created a benchmark by systematically injecting 36 different types of software "mutants" (bugs) into Apollo’s code to see if STRaP could still find them.
Key Insights and Ideas
- Leveraging the "Pub-Sub" Architecture: Most modern ADSs use a middleware (like ROS or CyberRT) where modules talk to each other. STRaP realizes that the most useful information for testing isn't in the raw video, but in the messages passed between these modules.
- The Rarity Heuristic: In software testing, "corner cases" are where bugs hide. By mathematically defining "rarity" based on how often specific feature combinations appear in a log, STRaP successfully front-loads the most "dangerous" scenarios.
- Glitches Matter: The authors found that raw DL model outputs are noisy. Without their "smoothing" step, the system would treat every 0.1-second sensor flicker as a new "scenario," defeating the purpose of data reduction.
Key Results
- Massive Time Savings: STRaP reduced the length of driving recordings by 34% to 77%, depending on the complexity of the map and the specific module being tested.
- High Fidelity: Despite the massive reduction, the system maintained a 99% fault detection rate. It caught almost every bug that the original, full-length recording would have caught.
- Faster Bug Discovery: The prioritization method improved the speed of fault detection (measured by APFD) by 22% to 53% compared to standard baselines like chronological or random ordering.
Limitations and Future Work
While highly effective, STRaP currently relies on a manually defined schema to tell it which features (like traffic lights) to look for. The authors suggest that future versions could use unsupervised machine learning to automatically discover "interesting" features without human intervention.
Additionally, while the study used high-fidelity simulators and real-world map layouts, the next step would be validating the approach on logs collected from physical vehicles on actual roads, where sensor noise and environmental unpredictability are even higher.
Summary updated