Abstract
While modern dialogue systems heavily rely on large language models (LLMs), their implementation often goes beyond pure LLM interaction. Developers integrate multiple LLMs, external tools, and databases. Therefore, assessment of the underlying LLM alone does not suffice, and the dialogue systems must be tested and evaluated as a whole. However, this remains a major challenge. With most previous work focusing on turn-level analysis, less attention has been paid to integrated dialogue-level quality assurance. To address this, we present ChatChecker, a framework for automated evaluation and testing of complex dialogue systems. ChatChecker uses LLMs to simulate diverse user interactions, identify dialogue breakdowns, and evaluate quality. Compared to previous approaches, our design reduces setup effort and is generalizable, as it does not require reference dialogues and is decoupled from the implementation of the target dialogue system. We improve breakdown detection performance over a prior LLM-based approach by including an error taxonomy in the prompt. Additionally, we propose a novel non-cooperative user simulator based on challenging personas that uncovers weaknesses in target dialogue systems more effectively. Through this, ChatChecker contributes to thorough and scalable testing. This enables both researchers and practitioners to accelerate the development of robust dialogue systems.
Introduction
Evaluating modern dialogue systems is a moving target. These systems are no longer just standalone Large Language Models (LLMs); they are complex architectures integrating multiple models, external databases, and API tools. Consequently, testing the underlying LLM in isolation is insufficient; the integrated system must be evaluated as a whole.
Current evaluation methods suffer from three primary limitations: a heavy dependency on "gold standard" human reference dialogues, a narrow focus on turn-level metrics rather than conversational flow, and a "cooperative user" bias. Most existing simulators act as perfect users, whereas real-world users can be impatient, contradictory, or confusing, leading to "dialogue breakdowns" where the system fails to maintain the conversation's logic or utility.
The ChatChecker Framework
ChatChecker introduces a modular, four-stage pipeline designed to automate the testing of complex dialogue systems without requiring reference datasets.
- Persona Generation: An LLM generates diverse user profiles, including Standard Personas and Challenging Personas (non-cooperative users who are impolite or prone to changing their minds).
- User Simulation: These personas drive conversations with the target system. The Challenging Persona Simulator (Chall-PerSim) is specifically designed to uncover edge cases.
- Breakdown Detection: An LLM acts as a judge, scanning dialogues for breakdowns. Performance is enhanced by providing the LLM with an extended error taxonomy (e.g., task-success impairments, efficiency issues) to guide its reasoning.
- Dialogue Rating: The framework provides dialogue-level ratings across dimensions like coherence and informativeness using calibrated 1-5 scales.
Key Results and Insights
Stress Testing via Non-cooperative Simulation
The most significant finding is the utility of the non-cooperative simulator. Challenging personas triggered substantially more unique error types and even caused target systems to "crash" (generate invalid formats) in a significant portion of test dialogues.
As shown in the experimental results, the Challenging Persona Simulator elicited significantly more breakdowns per turn (0.23 vs. 0.13) compared to standard simulators, proving to be a more effective adversarial testing tool. In the AutoTOD system, Chall-PerSim led to an average of 4.00 crashes, compared to only 1.60 for standard personas.
Detection Performance and Human Alignment
ChatChecker’s breakdown detector outperformed previous LLM-based baselines, achieving an F1 score of 0.764 on English data and 0.832 on Japanese data. Furthermore, the Dialogue Rater showed a strong positive correlation (ρ = 0.683) with human judgments on the FED-Dial chit-chat dataset, although correlation was lower on highly specific task-oriented datasets.
Conclusion
ChatChecker provides a scalable, zero-shot framework for testing dialogue systems by focusing on non-cooperative user interactions. By forcing LLMs to classify specific error types within a structured taxonomy, the framework achieves higher recall in breakdown detection. While error type classification remains a challenge—with exact match rates for specific error types at approximately 18.8%—the framework's ability to identify that a breakdown occurred is a significant step forward for robust system development. Future work will focus on incorporating safety evaluations and improving the granularity of rating dimensions.
Summary updated