Testing of autonomous driving systems: where are we and where should we go?

Guannan Lou, Yao Deng, Xi Zheng, Mengshi Zhang, Tianyi Zhang

TL;DR

This paper provides a reality check for the autonomous driving system (ADS) research community by bridging the gap between academic innovation and industrial practice. Through interviews and surveys with over 100 industry professionals, the authors identify a significant disconnect: while researchers focus heavily on testing individual AI models, practitioners are struggling with system-level complexities, massive data labeling bottlenecks, and the need for more sophisticated simulation tools.

What Problem Does This Paper Address?

Autonomous driving systems are not just "AI on wheels"; they are complex hybrids of deep learning (DL) models and traditional logic-based control modules. While the Software Engineering (SE) community has produced a flurry of papers on ADS testing, it hasn't been clear if these techniques actually solve the problems faced by engineers at companies like Waymo, Tesla, or Baidu.

The authors set out to answer three core questions:

  1. What are the actual industrial practices for testing ADSs?
  2. What are the emerging needs that current tools don't meet?
  3. To what extent does existing academic research address these industrial needs?

Methods

The researchers employed a "mixed-methods" approach to ensure both depth and breadth:

  • Semi-structured Interviews: They spoke with 10 developers from 10 different autonomous driving companies to gather qualitative insights.
  • Large-scale Survey: They surveyed 100 practitioners (out of nearly 2,000 invited) to quantitatively validate the interview findings.
  • Literature Review: They manually reviewed 117 papers from 28 top-tier SE conferences, identifying 42 specifically focused on ADS testing to build a comprehensive taxonomy of the current research landscape.

Datasets

While the paper does not introduce a new software dataset, it analyzes the use of existing ones. It notes that 54% of practitioners rely on public benchmarks like CityScapes, ApolloScape, and the Waymo Open Dataset. The authors have also made their own qualitative "Code Book" and survey results publicly available to encourage further research into practitioner needs.

Key Insights and Ideas

The study revealed several "aha moments" regarding the state of the industry:

  • Multi-Module vs. End-to-End: Academic research often focuses on "End-to-End" (E2E) models (where one giant neural network handles everything from camera input to steering). However, 100% of interviewees and 69% of survey respondents work on multi-module systems (separate modules for perception, prediction, and planning). E2E models are considered too hard to debug and prone to overfitting.
  • The "Consistency" Metric: In industry, model accuracy isn't the only king. Practitioners use custom metrics like consistency—for example, a lane detection model shouldn't wildly change its prediction between two consecutive video frames.
  • The Simulation Bottleneck: While simulation is safer and cheaper than on-road testing, it’s still too slow. To prove a system is "safe," it needs roughly $10^7$ hours of testing. Even with simulators running 10x faster than real-time, the computational cost is staggering.
  • The Data Labeling Nightmare: This was identified as the most urgent need. Engineers are drowning in raw sensor data (Lidar, Radar, Cameras) that must be manually labeled and synchronized by timestamp before it can be used for testing.

Key Results

The authors identified 7 Common Practices (e.g., using simulation for regression testing, designing scenarios based on crash reports) and 4 Emerging Needs:

  1. Identifying Corner Cases: Finding the "unknown unknowns" (e.g., how a car reacts to a sunset that blinds the sensors).
  2. Speeding up Testing: Moving beyond simple simulation to intelligent test selection and prioritization.
  3. Complex Scenario Construction: Current tools require hundreds of lines of code to describe a simple "cut-in" maneuver; industry needs higher-level abstractions.
  4. Data Labeling Support: Tools to automatically analyze, recognize, and repair errors in labeled data.

The Gap: The literature review showed that while research is very active in "Corner Case Generation" (Need 1), the other three needs—especially data labeling and scenario construction—are significantly under-investigated.

Limitations and Future Work

The authors acknowledge a few limitations: the survey response rate was low (5%), and industry participants may have been guarded due to confidentiality agreements.

Future Directions: * Multi-Objective Optimization: Researchers should develop testing techniques that maximize coverage across multiple modules (e.g., neural coverage for perception and code coverage for logic) simultaneously. * NLP for Scenarios: Using Natural Language Processing to automatically turn police crash reports or text descriptions into executable simulation scenarios. * Test Reduction: Applying classic SE "test suite reduction" techniques to the world of ADS to help hit that $10^7$-hour safety mark faster.

Summary updated

Abstract

Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners’ needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.
BibTeX
@inproceedings{lou2022testing,
  author = {Guannan Lou and Yao Deng and Xi Zheng and Mengshi Zhang and Tianyi Zhang},
  title = {{Testing of autonomous driving systems: where are we and where should we go?}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  url = {https://openreview.net/forum?id=3pT_nKt3Hc},
}

Loading related content...

Figure