Query as Test: An Intelligent Driving Test and Data Storage Method for Integrated Cockpit-Vehicle-Road Scenarios

Shengyue Yao, Runqing Guo, Yangyang Qin, Miangbing Meng, Jipeng Cao, Yilun Lin, Yisheng Lv, Fei-Yue WangarXiv 2025 · DOI · arXiv

Abstract

This paper introduces Query as Test (QaT), a paradigm shift in autonomous vehicle (AV) validation that moves away from rigid, pre-scripted test cases toward flexible, natural language logical queries. By representing heterogeneous data from vehicles, cockpits, and roads as a unified "knowledge base" using Answer Set Programming (ASP), the proposed framework enables developers to perform complex "what-if" analysis and formal safety checks with unprecedented speed and interpretability. Experimental results show that the framework achieves a 97.0% success rate in semantic query execution, significantly outperforming traditional LLM and database approaches.

1. Introduction

Current autonomous driving (AD) testing relies heavily on "data stacking"—the collection of massive volumes of driving data for replay in simulations to identify system failures. This approach faces three critical challenges:

  1. Inflexibility: Identifying specific, complex scenarios (e.g., a distracted driver encountering a jaywalker in rain) within petabytes of raw data is labor-intensive.
  2. Data Fragmentation: Information from intelligent cockpits, vehicle sensors, and road infrastructure (V2X) often resides in incompatible silos.
  3. The Test Oracle Problem: Defining "pass/fail" criteria is frequently reduced to simple numerical thresholds, such as Time-to-Collision (TTC), which fail to capture the nuances of safe and comfortable driving.

2. Methodology

The authors propose a two-part solution to unify data representation and simplify testing procedures.

2.1 Extensible Scenarios Notations (ESN)

Rather than utilizing traditional relational databases, the framework "ASP-ifies" driving data. Using Answer Set Programming (ASP), a declarative logic programming language, driving scenes are represented as a collection of facts (e.g., holds(position(car_01, X, Y), T)) and logical rules. This creates a unified semantic "glue" that connects driver state, vehicle physics, and road topology into a single queryable logic base.

2.2 Query as Test (QaT) & Neuro-Symbolic Integration

A neuro-symbolic approach bridges the gap between human intent and formal logic. A Large Language Model (LLM) front-end deconstructs natural language queries into structured safety assertions, which are then translated into formal ASP queries for execution by a deterministic logic engine.

The Query as Test (QaT) workflow and neuro-symbolic translation engine.

This architecture allows for counterfactual analysis, enabling developers to modify logic rules (e.g., adjusting braking aggressiveness) and observe results without the need for computationally expensive physics re-simulations.

3. Experimental Evaluation

The framework was evaluated using the Waymo Open Motion Dataset (WOMD) and nuScenes, alongside a custom benchmark of 30 distinct scenarios covering urban, highway, and edge-case conditions.

3.1 Performance Metrics

The ESN-QaT framework was compared against traditional SQL-based search, Retrieval-Augmented Generation (RAG), and direct LLM querying.

Comparison of success rates across different query methods.

As shown in the results, ESN-QaT achieved a 97.0% success rate in answering complex semantic queries, far exceeding Direct LLM (84.5%), RAG (49.7%), and SQL (9.2%). Furthermore, the system demonstrated high efficiency with an average response time of 40ms, compared to over 3.6 seconds for LLM-based approaches. The deterministic nature of the logic engine also yielded an accuracy score of 0.951, mitigating the hallucination risks inherent in probabilistic models.

4. Discussion

4.1 Privacy through Abstraction

ESN facilitates "on-demand data abstraction." Instead of sharing raw GPS coordinates, data owners can share logical facts like holds(in_region(vehicle_A, "downtown_la"), T). This maintains utility for traffic analysis while protecting individual privacy.

4.2 Interpretability

Unlike "black-box" AI models, every query result in this framework includes a formal proof trace. If a safety violation is detected, the system identifies the exact logical facts and timestamps responsible for the failure, providing verifiable evidence for developers and regulators.

5. Conclusion and Future Work

While the ESN-QaT framework offers significant advantages in flexibility and interpretability, scalability remains a primary technical hurdle. The "grounding" process for logic rules can become computationally expensive for industrial-scale datasets. Future research will focus on hybrid architectures combining ASP with traditional databases and the use of Graph Neural Networks for automatic rule generation to further align with standards such as the EU AI Act and ISO 26262.

Summary updated

Abstract

With the deep penetration of Artificial Intelligence (AI) in the transportation sector, intelligent cockpits, autonomous driving, and intelligent road networks are developing at an unprecedented pace. However, the data ecosystems of these three key areas are increasingly fragmented and incompatible. Especially, existing testing methods rely on data stacking, fail to cover all edge cases, and lack flexibility. To address this issue, this paper introduces the concept of "Query as Test" (QaT). This concept shifts the focus from rigid, prescripted test cases to flexible, on-demand logical queries against a unified data representation. Specifically, we identify the need for a fundamental improvement in data storage and representation, leading to our proposal of "Extensible Scenarios Notations" (ESN). ESN is a novel declarative data framework based on Answer Set Programming (ASP), which uniformly represents heterogeneous multimodal data from the cockpit, vehicle, and road as a collection of logical facts and rules. This approach not only achieves deep semantic fusion of data, but also brings three core advantages: (1) supports complex and flexible semantic querying through logical reasoning; (2) provides natural interpretability for decision-making processes; (3) allows for on-demand data abstraction through logical rules, enabling fine-grained privacy protection. We further elaborate on the QaT paradigm, transforming the functional validation and safety compliance checks of autonomous driving systems into logical queries against the ESN database, significantly enhancing the expressiveness and formal rigor of the testing. Finally, we introduce the concept of "Validation-Driven Development" (VDD), which suggests to guide developments by logical validation rather than quantitative testing in the era of Large Language Models, in order to accelerating the iteration and development process.
BibTeX
@misc{yao2025query,
  author = {Shengyue Yao and Runqing Guo and Yangyang Qin and Miangbing Meng and Jipeng Cao and Yilun Lin and Yisheng Lv and Fei-Yue Wang},
  title = {{Query as Test: An Intelligent Driving Test and Data Storage Method for Integrated Cockpit-Vehicle-Road Scenarios}},
  year = {2025},
  eprint = {2506.22068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.22068},
}
Figure