Abstract
This paper introduces Query as Test (QaT), a paradigm shift in autonomous vehicle (AV) validation that moves away from rigid, pre-scripted test cases toward flexible, natural language logical queries. By representing heterogeneous data from vehicles, cockpits, and roads as a unified "knowledge base" using Answer Set Programming (ASP), the proposed framework enables developers to perform complex "what-if" analysis and formal safety checks with unprecedented speed and interpretability. Experimental results show that the framework achieves a 97.0% success rate in semantic query execution, significantly outperforming traditional LLM and database approaches.
1. Introduction
Current autonomous driving (AD) testing relies heavily on "data stacking"—the collection of massive volumes of driving data for replay in simulations to identify system failures. This approach faces three critical challenges:
- Inflexibility: Identifying specific, complex scenarios (e.g., a distracted driver encountering a jaywalker in rain) within petabytes of raw data is labor-intensive.
- Data Fragmentation: Information from intelligent cockpits, vehicle sensors, and road infrastructure (V2X) often resides in incompatible silos.
- The Test Oracle Problem: Defining "pass/fail" criteria is frequently reduced to simple numerical thresholds, such as Time-to-Collision (TTC), which fail to capture the nuances of safe and comfortable driving.
2. Methodology
The authors propose a two-part solution to unify data representation and simplify testing procedures.
2.1 Extensible Scenarios Notations (ESN)
Rather than utilizing traditional relational databases, the framework "ASP-ifies" driving data. Using Answer Set Programming (ASP), a declarative logic programming language, driving scenes are represented as a collection of facts (e.g., holds(position(car_01, X, Y), T)) and logical rules. This creates a unified semantic "glue" that connects driver state, vehicle physics, and road topology into a single queryable logic base.
2.2 Query as Test (QaT) & Neuro-Symbolic Integration
A neuro-symbolic approach bridges the gap between human intent and formal logic. A Large Language Model (LLM) front-end deconstructs natural language queries into structured safety assertions, which are then translated into formal ASP queries for execution by a deterministic logic engine.
This architecture allows for counterfactual analysis, enabling developers to modify logic rules (e.g., adjusting braking aggressiveness) and observe results without the need for computationally expensive physics re-simulations.
3. Experimental Evaluation
The framework was evaluated using the Waymo Open Motion Dataset (WOMD) and nuScenes, alongside a custom benchmark of 30 distinct scenarios covering urban, highway, and edge-case conditions.
3.1 Performance Metrics
The ESN-QaT framework was compared against traditional SQL-based search, Retrieval-Augmented Generation (RAG), and direct LLM querying.
As shown in the results, ESN-QaT achieved a 97.0% success rate in answering complex semantic queries, far exceeding Direct LLM (84.5%), RAG (49.7%), and SQL (9.2%). Furthermore, the system demonstrated high efficiency with an average response time of 40ms, compared to over 3.6 seconds for LLM-based approaches. The deterministic nature of the logic engine also yielded an accuracy score of 0.951, mitigating the hallucination risks inherent in probabilistic models.
4. Discussion
4.1 Privacy through Abstraction
ESN facilitates "on-demand data abstraction." Instead of sharing raw GPS coordinates, data owners can share logical facts like holds(in_region(vehicle_A, "downtown_la"), T). This maintains utility for traffic analysis while protecting individual privacy.
4.2 Interpretability
Unlike "black-box" AI models, every query result in this framework includes a formal proof trace. If a safety violation is detected, the system identifies the exact logical facts and timestamps responsible for the failure, providing verifiable evidence for developers and regulators.
5. Conclusion and Future Work
While the ESN-QaT framework offers significant advantages in flexibility and interpretability, scalability remains a primary technical hurdle. The "grounding" process for logic rules can become computationally expensive for industrial-scale datasets. Future research will focus on hybrid architectures combining ASP with traditional databases and the use of Graph Neural Networks for automatic rule generation to further align with standards such as the EU AI Act and ISO 26262.
Summary updated