GCDF1: A Goal- and Context- Driven F-Score for Evaluating User Models

Alexandru Coca, Bo-Hsiang Tseng, Bill Byrne2021 · DOI

Abstract

The evaluation of dialogue systems in interaction with simulated users has been proposed to improve turn-level, corpus-based metrics which can only evaluate test cases encountered in a corpus and cannot measure system's ability to sustain multi-turn interactions. Recently, little emphasis was put on automatically assessing the quality of the user model itself, so unless correlations with human studies are measured, the reliability of user model based evaluation is unknown. We propose GCDF1, a simple but effective measure of the quality of semantic-level conversations between a goal-driven user agent and a system agent. In contrast with previous approaches we measure the F-score at dialogue level and consider user and system behaviours to improve recall and precision estimation. We facilitate scores interpretation by providing a rich hierarchical structure with information about conversational patterns present in the test data and tools to efficiently query the conversations generated. We apply our framework to assess the performance and weaknesses of a Convlab2 user model.

Introduction

To build better dialogue systems, researchers often rely on simulated users to conduct large-scale testing. However, if these simulators do not behave like real humans, the resulting evaluations are unreliable. Traditional metrics are often turn-level, comparing a single simulator response to a static human response. This approach fails to account for the dynamic nature of dialogue, where a simulator might provide a valid but unexpected response, or naturally repeat information to ensure system understanding. We introduce GCDF1 (Goal- and Context-Driven F-Score) to provide a dialogue-level evaluation that accounts for these nuances.

Methodology

GCDF1 evaluates the quality of semantic-level interactions by splitting the assessment into two parts: I-GCDF1 (Informs) and R-GCDF1 (Requests). Unlike turn-based metrics, GCDF1 looks at the recall and precision of goals across the entire session.

Key innovations include:

  • Context-Awareness: The algorithm distinguishes between "intrinsic" repetition (model glitches) and "elicited" repetition (system misunderstanding). It analyzes the span between repetitions to avoid penalizing the user model for system-driven errors.
  • Pre-emption Detection: It recognizes when a system provides information before the user asks for it, adjusting scores so the user model isn't penalized for "missing" a request that is no longer necessary.
  • Value Normalization: Fuzzy matching (Levenshtein distance) is used to map synonyms (e.g., "cheap" vs. "low price") to canonical forms, ensuring robust evaluation.

Experimental Setup and Insights

We evaluated user models and system agents within the ConvLab-2 framework using the MultiWOZ 2.1 dataset. Our study highlighted an "efficiency bias" in current models, which tend to provide all constraints in a single turn rather than spreading them out as humans do.

Distribution of dialogues across domains for information and request goals.

Results and Diagnostic Analysis

Applying GCDF1 revealed significant performance variance across domains. For instance, I-GCDF1 was high for "Attraction" (0.786) but much lower for "Restaurant" (0.467) and "Taxi" (0.482). The metric also demonstrated strong diagnostic power; in the "Train" domain, it identified that the day slot was repeated in nearly every dialogue due to the system's booking confirmation being misidentified as a request.

Breakdown of dialogue failure types across different domains, highlighting NLU failures and domain switches.

As shown in Figure 2, specific errors such as NLU failures and domain switches frequently derail the dialogue chain. The high number of "unmatched" dialogues in the restaurant and hotel domains correlates with the lower I-GCDF1 scores observed, providing researchers with clear targets for model improvement.

Conclusion

GCDF1 offers a more realistic and diagnostic measure of user simulator quality by focusing on dialogue-level goals and context. While currently limited to semantic-level dialogue acts, future work will extend this framework to natural language text and more complex datasets like Google’s Schema-Guided Dialogue (SGD).

Summary updated

Abstract

The evaluation of dialogue systems in interaction with simulated users has been proposed to improve turn-level, corpus-based metrics which can only evaluate test cases encountered in a corpus and cannot measure system's ability to sustain multi-turn interactions. Recently, little emphasis was put on automatically assessing the quality of the user model itself, so unless correlations with human studies are measured, the reliability of user model based evaluation is unknown. We propose GCDF1, a simple but effective measure of the quality of semantic-level conversations between a goaldriven user agent and a system agent. In contrast with previous approaches we measure the F-score at dialogue level and consider user and system behaviours to improve recall and precision estimation. We facilitate scores interpretation by providing a rich hierarchical structure with information about conversational patterns present in the test data and tools to efficiently query the conversations generated. We apply our framework to assess the performance and weaknesses of a Convlab2 user model 1 .
BibTeX
@misc{coca2021gcdf,
  author = {Alexandru Coca and Bo-Hsiang Tseng and Bill Byrne},
  title = {{GCDF1: A Goal- and Context- Driven F-Score for Evaluating User Models}},
  year = {2021},
  doi = {10.18653/v1/2021.eancs-1.2},
}
Figure