Abstract
The evaluation of dialogue systems in interaction with simulated users has been proposed to improve turn-level, corpus-based metrics which can only evaluate test cases encountered in a corpus and cannot measure system's ability to sustain multi-turn interactions. Recently, little emphasis was put on automatically assessing the quality of the user model itself, so unless correlations with human studies are measured, the reliability of user model based evaluation is unknown. We propose GCDF1, a simple but effective measure of the quality of semantic-level conversations between a goal-driven user agent and a system agent. In contrast with previous approaches we measure the F-score at dialogue level and consider user and system behaviours to improve recall and precision estimation. We facilitate scores interpretation by providing a rich hierarchical structure with information about conversational patterns present in the test data and tools to efficiently query the conversations generated. We apply our framework to assess the performance and weaknesses of a Convlab2 user model.
Introduction
To build better dialogue systems, researchers often rely on simulated users to conduct large-scale testing. However, if these simulators do not behave like real humans, the resulting evaluations are unreliable. Traditional metrics are often turn-level, comparing a single simulator response to a static human response. This approach fails to account for the dynamic nature of dialogue, where a simulator might provide a valid but unexpected response, or naturally repeat information to ensure system understanding. We introduce GCDF1 (Goal- and Context-Driven F-Score) to provide a dialogue-level evaluation that accounts for these nuances.
Methodology
GCDF1 evaluates the quality of semantic-level interactions by splitting the assessment into two parts: I-GCDF1 (Informs) and R-GCDF1 (Requests). Unlike turn-based metrics, GCDF1 looks at the recall and precision of goals across the entire session.
Key innovations include:
- Context-Awareness: The algorithm distinguishes between "intrinsic" repetition (model glitches) and "elicited" repetition (system misunderstanding). It analyzes the span between repetitions to avoid penalizing the user model for system-driven errors.
- Pre-emption Detection: It recognizes when a system provides information before the user asks for it, adjusting scores so the user model isn't penalized for "missing" a request that is no longer necessary.
- Value Normalization: Fuzzy matching (Levenshtein distance) is used to map synonyms (e.g., "cheap" vs. "low price") to canonical forms, ensuring robust evaluation.
Experimental Setup and Insights
We evaluated user models and system agents within the ConvLab-2 framework using the MultiWOZ 2.1 dataset. Our study highlighted an "efficiency bias" in current models, which tend to provide all constraints in a single turn rather than spreading them out as humans do.
Results and Diagnostic Analysis
Applying GCDF1 revealed significant performance variance across domains. For instance, I-GCDF1 was high for "Attraction" (0.786) but much lower for "Restaurant" (0.467) and "Taxi" (0.482). The metric also demonstrated strong diagnostic power; in the "Train" domain, it identified that the day slot was repeated in nearly every dialogue due to the system's booking confirmation being misidentified as a request.
As shown in Figure 2, specific errors such as NLU failures and domain switches frequently derail the dialogue chain. The high number of "unmatched" dialogues in the restaurant and hotel domains correlates with the lower I-GCDF1 scores observed, providing researchers with clear targets for model improvement.
Conclusion
GCDF1 offers a more realistic and diagnostic measure of user simulator quality by focusing on dialogue-level goals and context. While currently limited to semantic-level dialogue acts, future work will extend this framework to natural language text and more complex datasets like Google’s Schema-Guided Dialogue (SGD).
Summary updated