Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Tomáš Nekvinda, Ondřej DušekACL 2021

TL;DR

This paper highlights significant inconsistencies in how the popular MultiWOZ dialogue dataset is evaluated, leading to unreliable comparisons of system performance. The authors propose standardized evaluation scripts and advocate for greater transparency in reporting, offering a more robust foundation for future research in task-oriented dialogue systems.

What Problem Does This Paper Address?

Task-oriented dialogue systems aim to help users achieve specific goals, like booking a flight or finding a restaurant. The MultiWOZ dataset is a widely used benchmark for evaluating these systems, particularly their ability to generate contextually appropriate responses. However, the paper identifies a critical issue: the metrics used to evaluate these systems, especially BLEU (a measure of text similarity) and Inform & Success rates (measuring task completion), are implemented inconsistently across different research papers. This inconsistency stems from variations in data preprocessing (like delexicalization, where specific details are replaced with placeholders) and metric calculation. Consequently, reported scores are not directly comparable, obscuring genuine progress and making it difficult to trust state-of-the-art claims.

Methods

The core of this paper's contribution lies in its rigorous analysis and standardization efforts. The authors:

  • Identify and analyze inconsistencies: They meticulously examine how various systems preprocess data and calculate metrics like BLEU, Inform, and Success rates on the MultiWOZ dataset.
  • Re-evaluate existing systems: They re-run 13 existing end-to-end and policy optimization models using their own standardized scripts and preprocessing pipeline. This allows for a fair, apples-to-apples comparison.
  • Develop standardized evaluation scripts: They release a set of unified scripts that implement delexicalization, metric calculation, and postprocessing in a consistent manner. This aims to be the new standard for evaluating on MultiWOZ.
  • Propose diversity metrics: Beyond traditional metrics, they also evaluate system outputs for diversity, arguing it's an important aspect of naturalness and long-term usability.

Datasets

  • MultiWOZ 2.0 and 2.1: The primary dataset used for evaluation. The paper discusses the evolution of the dataset, including MultiWOZ 2.1's improvements in state annotation and explicit system action annotation, and MultiWOZ 2.2's further fixes and ontology refinements.

Key Insights and Ideas

  • Delexicalization is a major source of variance: Different methods of replacing specific slot values (like names or addresses) with placeholders lead to significantly different BLEU scores, even for the same system. This highlights the need for a standardized delexicalization approach.
  • Inform & Success rates are highly sensitive to implementation: The way database queries are handled, how offered entities are matched against goals, and even how active domains are inferred can drastically alter Inform and Success scores.
  • Reported scores are often optimistic: The authors found that many reported scores for existing systems were higher than what they achieved with their standardized setup, suggesting that some systems might be exploiting inconsistencies in evaluation rather than demonstrating true performance gains.
  • Output diversity matters: Traditional metrics don't capture how repetitive or formulaic a system's responses might be, which is crucial for user experience.

Key Results

  • The paper demonstrates that different delexicalization styles can change BLEU scores by up to 2% absolute, and significantly alter the relative ranking of systems.
  • When re-evaluating existing systems with their standardized scripts, the authors found that many reported Inform and Success rates were considerably higher than their own findings, sometimes by as much as 20%.
  • They provide a comprehensive comparison of 13 systems across various delexicalization styles and metric implementations, revealing substantial discrepancies.
  • The authors release their standardized evaluation scripts and system outputs, enabling future researchers to conduct more reliable comparisons.

Limitations and Future Work

  • Dataset limitations: The paper notes that the MultiWOZ dataset has limitations, including potential issues with database surface forms, lack of booking availability information, and a non-trivial overlap between training and test data folds, which might allow systems to "memorize" rather than generalize.
  • Need for human evaluation: While corpus-based metrics are useful, the authors reiterate the irreplaceable value of human judgment and full dialogue evaluation, which is still not standard practice for end-to-end dialogue systems.
  • Future work: The authors suggest a manual revision of the MultiWOZ 2.2 span annotation to improve lexicalized output evaluation and strongly advocate for the increased use of human evaluation alongside automated metrics.

Summary updated

BibTeX
@inproceedings{nekvinda2021shades,
  author = {Tomáš Nekvinda and Ondřej Dušek},
  title = {{Shades of BLEU, Flavours of Success: The Case of MultiWOZ}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure