An investigation into the state-of-the-practice autonomous driving testing

Guannan Lou, Yao Deng, Xi Zheng, Tianyi Zhang, Mengshi Zhang

Abstract

Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners' needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.

Introduction

Testing an Autonomous Driving System (ADS) presents unique challenges compared to traditional software. These "cyber-physical" systems integrate deep learning (DL) for perception with logic-based code for planning and control. While academic research in Software Engineering (SE) has produced numerous techniques for ADS testing, such as metamorphic testing for environmental simulations, there remains a lack of clarity on whether these solutions align with industrial realities. This study, presented at ESEC/FSE 2022, investigates the current state-of-the-practice to identify industrial needs and evaluate the extent to which current research addresses them.

Methodology

The researchers employed a mixed-methods approach to ensure both depth and breadth in their findings. The study was conducted in three primary stages:

  1. Semi-structured Interviews: Qualitative insights were gathered from 10 experienced developers across 10 different ADS companies.
  2. Large-scale Survey: Findings from the interviews were quantitatively validated through a survey of 100 ADS practitioners.
  3. Systematic Literature Review: A taxonomy of existing research was built by analyzing 42 papers from top-tier SE conferences, identifying critical gaps between academia and industry.
Overview of the three-step research design including interviews, surveys, and literature review.

Current Industrial Practices

The study identified several key characteristics of how ADSs are tested in industry:

  • Multi-Module Architectures: Unlike many academic models that favor "End-to-End" (E2E) approaches, 100% of interviewees utilized multi-module systems. Modularity is preferred in industry for its debuggability and resistance to overfitting.
  • System-Level Metrics: While researchers often focus on model accuracy, practitioners prioritize system-level metrics such as consistency (e.g., stable lane detection) and latency (reaction speed).
  • Diverse Testing Scenarios: Testing encompasses a wide range of scenarios, from basic car-following to complex maneuvers like turning and lane changes. These are conducted in both real-world and simulation environments, often leveraging industrial benchmarks like CityScapes, ApolloScape, and the Waymo Open Dataset.
Comparison of testing scenarios across real-world and simulation environments.

The Research-Practice Gap

A significant disconnect exists between academic focus and industrial needs. While both groups prioritize corner case generation, several critical industrial needs remain underserved by the SE community:

  1. Simulation Speed: Industry requires "test reduction" and "prioritization" techniques to overcome the massive time requirements of simulation-based testing.
  2. Scenario Construction: Practitioners need higher-level abstractions for scenario description, as current methods (like OpenScenario) are often too verbose.
  3. Data Labeling: Tool support for efficient and high-quality data labeling was identified as a top priority (68% importance), yet it remains largely ignored in SE research.

Conclusion and Future Directions

The study highlights the need for researchers to move beyond simple image perturbations toward multi-objective optimization for complex, multi-module systems. Future work should explore automating the conversion of real-world data, such as police crash reports, into executable simulation scenarios. By addressing these critical bottlenecks like data labeling and simulation acceleration, the SE community can better support the development of safe and reliable autonomous driving systems.

Summary updated

Abstract

Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners' needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.
BibTeX
@inproceedings{lou2022investigation,
  author = {Guannan Lou and Yao Deng and Xi Zheng and Tianyi Zhang and Mengshi Zhang},
  title = {{An investigation into the state-of-the-practice autonomous driving testing}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  url = {https://openreview.net/forum?id=3pT_nKt3Hc},
}
Figure