Abstract
Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners' needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.
Introduction
Autonomous driving systems (ADSs) are not just "AI on wheels"; they are complex hybrids of deep learning (DL) models and traditional logic-based control modules. While the Software Engineering (SE) community has produced a flurry of research on ADS testing, a significant gap remains between academic innovation and industrial practice. This study aims to bridge that gap by answering three core questions:
- What are the actual industrial practices for testing ADSs?
- What are the emerging needs that current tools don't meet?
- To what extent does existing academic research address these industrial needs?
Methodology
The researchers employed a mixed-methods approach to ensure both depth and breadth in their findings, involving qualitative interviews, quantitative surveys, and a systematic literature review.
- Semi-structured Interviews: Qualitative insights were gathered from 10 developers across 10 different autonomous driving companies.
- Large-scale Survey: 100 practitioners validated the interview findings quantitatively.
- Literature Review: A review of 42 SE papers specifically focused on ADS testing was conducted to build a comprehensive taxonomy of the current research landscape.
Key Industrial Insights
The study revealed several "aha moments" regarding the state of the industry and its divergence from academic focus:
- Multi-Module vs. End-to-End: While research often focuses on "End-to-End" (E2E) models, 100% of interviewees work on multi-module systems (separate modules for perception, prediction, and planning). E2E models are considered too hard to debug and prone to overfitting in industrial contexts.
- The "Consistency" Metric: In industry, model accuracy isn't the only priority. Practitioners use custom metrics like consistency—ensuring a model doesn't wildly change predictions between consecutive video frames.
- The Simulation Bottleneck: Proving a system is "safe" requires roughly $10^7$ hours of testing. Even with simulators running 10x faster than real-time, the computational cost is staggering.
- The Data Labeling Nightmare: This was identified as the most urgent need. Engineers are drowning in raw sensor data that must be manually labeled and synchronized before it can be used for testing.
Emerging Needs and Research Gaps
The authors identified 4 Emerging Needs that are critical to practitioners but often overlooked by current research. Survey data confirms that these needs are viewed as highly important by the developer community.
- Identifying Corner Cases: Finding "unknown unknowns," such as how a car reacts to a sunset that blinds sensors.
- Speeding up Testing: Moving beyond simple simulation to intelligent test selection and prioritization.
- Complex Scenario Construction: Developing higher-level abstractions to replace hundreds of lines of code for simple maneuvers.
- Data Labeling Support: Tools to automatically analyze, recognize, and repair errors in labeled data.
The Gap: The literature review showed that while research is very active in "Corner Case Generation" (Need 1), the other three needs—especially data labeling and scenario construction—are significantly under-investigated.
Future Directions
To better align research with industrial reality, the authors propose several directions for SE researchers:
- Multi-Objective Optimization: Develop testing techniques that maximize coverage across multiple modules (e.g., neural coverage for perception and code coverage for logic) simultaneously.
- NLP for Scenarios: Use Natural Language Processing to automatically turn police crash reports into executable simulation scenarios.
- Test Reduction: Apply classic SE "test suite reduction" techniques to help hit safety testing marks faster.
Summary updated