Abstract
In-car Conversational Question Answering (ConvQA) systems must navigate complex logical constraints beyond simple grammatical correctness. This paper introduces a benchmarking framework to evaluate the "contextual understanding" of these systems using Large Language Models (LLMs) as judges. Through an evaluation of 13 models and six prompting strategies, we demonstrate that reasoning models like DeepSeek-R1 can achieve near-perfect accuracy (0.99 F1-score) in identifying system failures, offering a scalable and cost-effective alternative to human evaluation.
Introduction
Modern in-car voice assistants are increasingly sophisticated, yet they often struggle with contextual constraints. A system might provide a linguistically sound recommendation that is practically useless—for instance, suggesting a restaurant that is currently closed or too far from the user's location. Traditional automated metrics like BLEU or ROUGE, which focus on word overlap, are insufficient for capturing these logical nuances. While human evaluation remains the gold standard, its high cost and lack of scalability hinder rapid development. This research explores the use of LLMs-as-a-Judge to automatically detect violations of user constraints such as time, distance, price, and cuisine.
Methods
The study evaluated 13 LLMs, ranging from 8B to 405B parameters, including specialized reasoning models. To optimize performance, the researchers employed six distinct prompting techniques designed to elicit deeper reasoning and reduce bias.
These strategies include standard Input-Output (I/O), Chain of Thought (CoT) for step-by-step reasoning, and more complex multi-agent approaches like Multi-Agent Debate (MAD) and Agent Roundtable (AR), where different models or personas deliberate to reach a consensus.
Datasets and Experimental Setup
In collaboration with BMW, a synthetic but human-validated dataset was developed, consisting of 600 recommendations based on 100 unique user "blocks" (context and utterance). Each request was paired with one correct response and five controlled "failure" responses. These failures targeted specific constraints: Location (distance), Time (opening hours), Cuisine (type), Cost (price range), and Rating (quality).
As shown in the example above, a system might fail by recommending a restaurant that is closed at the user's specific request time, despite meeting other criteria like cuisine type and rating. Human annotators validated the dataset's realism, giving it a score of 3.95/5.
Key Results
The evaluation revealed several critical insights into the capabilities of modern LLMs:
- Superiority of Reasoning Models: Specialized reasoning models, such as DeepSeek-R1 and OpenAI’s o3-mini, significantly outperformed traditional LLMs. DeepSeek-R1 achieved a peak F1-score of 0.99 using single-agent prompting with self-consistency.
- Prompting as a Performance Leveler: Advanced prompting techniques provided the most significant gains for smaller, non-reasoning models like Mistral-Nemo (12B), which reached an F1 of 0.714. However, top-tier reasoning models showed diminishing returns with complex prompts, as their base reasoning capabilities were already sufficient.
- Cost and Efficiency: DeepSeek-V3 emerged as a highly efficient option, leading the non-reasoning category with an F1-score of 0.985 while maintaining low latency and a cost of only $0.001 per request. In comparison, larger models like Llama-405B were significantly slower, taking up to 50 seconds per request.
Discussion: The "Cost of Luxury" Challenge
One notable finding was that models struggled most with Cost errors. LLMs frequently failed to infer that descriptors like "high-end" or "luxurious" imply a high-cost category. In some cases, models incorrectly judged a low-cost recommendation as valid because the price was not explicitly stated in currency, highlighting a need for more precise semantic mapping in prompts.
Conclusion and Future Work
This benchmarking framework demonstrates that LLMs can effectively serve as automated judges for in-car conversational systems, drastically reducing the need for human intervention. Future research will focus on expanding the framework to multi-turn interactions, broader domains like EV charging stations, and multilingual support to ensure global applicability for automotive deployments.
Summary updated