TL;DR
This paper provides a reality check for the autonomous driving system (ADS) research community by bridging the gap between academic innovation and industrial practice. Through interviews and surveys with over 100 industry professionals, the authors identify a significant disconnect: while researchers focus heavily on testing individual AI models, practitioners are struggling with system-level complexities, massive data labeling bottlenecks, and the need for more sophisticated simulation tools.
What Problem Does This Paper Address?
Autonomous driving systems are not just "AI on wheels"; they are complex hybrids of deep learning (DL) models and traditional logic-based control modules. While the Software Engineering (SE) community has produced a flurry of papers on ADS testing, it hasn't been clear if these techniques actually solve the problems faced by engineers at companies like Waymo, Tesla, or Baidu.
The authors set out to answer three core questions:
- What are the actual industrial practices for testing ADSs?
- What are the emerging needs that current tools don't meet?
- To what extent does existing academic research address these industrial needs?
Methods
The researchers employed a "mixed-methods" approach to ensure both depth and breadth:
- Semi-structured Interviews: They spoke with 10 developers from 10 different autonomous driving companies to gather qualitative insights.
- Large-scale Survey: They surveyed 100 practitioners (out of nearly 2,000 invited) to quantitatively validate the interview findings.
- Literature Review: They manually reviewed 117 papers from 28 top-tier SE conferences, identifying 42 specifically focused on ADS testing to build a comprehensive taxonomy of the current research landscape.
Datasets
While the paper does not introduce a new software dataset, it analyzes the use of existing ones. It notes that 54% of practitioners rely on public benchmarks like CityScapes, ApolloScape, and the Waymo Open Dataset. The authors have also made their own qualitative "Code Book" and survey results publicly available to encourage further research into practitioner needs.
Key Insights and Ideas
The study revealed several "aha moments" regarding the state of the industry:
- Multi-Module vs. End-to-End: Academic research often focuses on "End-to-End" (E2E) models (where one giant neural network handles everything from camera input to steering). However, 100% of interviewees and 69% of survey respondents work on multi-module systems (separate modules for perception, prediction, and planning). E2E models are considered too hard to debug and prone to overfitting.
- The "Consistency" Metric: In industry, model accuracy isn't the only king. Practitioners use custom metrics like consistency—for example, a lane detection model shouldn't wildly change its prediction between two consecutive video frames.
- The Simulation Bottleneck: While simulation is safer and cheaper than on-road testing, it’s still too slow. To prove a system is "safe," it needs roughly $10^7$ hours of testing. Even with simulators running 10x faster than real-time, the computational cost is staggering.
- The Data Labeling Nightmare: This was identified as the most urgent need. Engineers are drowning in raw sensor data (Lidar, Radar, Cameras) that must be manually labeled and synchronized by timestamp before it can be used for testing.
Key Results
The authors identified 7 Common Practices (e.g., using simulation for regression testing, designing scenarios based on crash reports) and 4 Emerging Needs:
- Identifying Corner Cases: Finding the "unknown unknowns" (e.g., how a car reacts to a sunset that blinds the sensors).
- Speeding up Testing: Moving beyond simple simulation to intelligent test selection and prioritization.
- Complex Scenario Construction: Current tools require hundreds of lines of code to describe a simple "cut-in" maneuver; industry needs higher-level abstractions.
- Data Labeling Support: Tools to automatically analyze, recognize, and repair errors in labeled data.
The Gap: The literature review showed that while research is very active in "Corner Case Generation" (Need 1), the other three needs—especially data labeling and scenario construction—are significantly under-investigated.
Limitations and Future Work
The authors acknowledge a few limitations: the survey response rate was low (5%), and industry participants may have been guarded due to confidentiality agreements.
Future Directions: * Multi-Objective Optimization: Researchers should develop testing techniques that maximize coverage across multiple modules (e.g., neural coverage for perception and code coverage for logic) simultaneously. * NLP for Scenarios: Using Natural Language Processing to automatically turn police crash reports or text descriptions into executable simulation scenarios. * Test Reduction: Applying classic SE "test suite reduction" techniques to the world of ADS to help hit that $10^7$-hour safety mark faster.
Summary updated