ChatEval: A Tool for Chatbot Evaluation

João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, Chris Callison-BurchNAACL 2019 · OpenReview

Abstract: João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, Chris Callison-Burch. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations).

2019.

Introduction

Evaluating open-domain chatbots is a significant challenge in NLP. Unlike tasks with objective ground truths, chatbot prompts can elicit an infinite variety of valid responses. This has led to a "Wild West" of evaluation characterized by poor correlation between automatic metrics and human perception, fragmented evaluation setups across research groups, and a reproducibility crisis where model code and parameters are rarely shared.

ChatEval is a unified, open-source framework and web portal designed to address these issues. By providing a standardized pipeline for both automatic metrics and crowdsourced human A/B testing, it enables researchers to compare dialog models transparently and reproducibly.

The ChatEval Framework

The authors introduce ChatEval as a two-part solution consisting of an open-source evaluation toolkit and a centralized web hub at chateval.org.

The ChatEval workflow: researchers submit models, which undergo automatic and human evaluation before results are shared publicly.

The web portal allows researchers to upload model responses to standardized prompts, serving as a repository for model code, parameters, and results. This centralization facilitates direct comparisons with prior work.

Human Evaluation Pipeline

A core component of ChatEval is its human evaluation pipeline. Rather than asking humans to provide subjective absolute ratings (e.g., a 1-5 scale), ChatEval employs A/B testing. Crowd workers on Amazon Mechanical Turk (AMT) are presented with a prompt and two different model responses. They must then select the better response or declare a tie.

The human evaluation interface on Amazon Mechanical Turk, showing an A/B comparison task.

This relative comparison method is more discriminative and less noisy than absolute ratings. The toolkit also computes standard automatic metrics such as lexical diversity (distinct-n), embedding-based cosine similarity, BLEU-2, and response perplexity. To ensure statistical rigor, the authors use Item Response Theory (IRT) to identify which prompts are most effective at distinguishing between high- and low-quality models.

Standardized Datasets

To ensure consistency across evaluations, ChatEval provides several curated sets:

  • DBDC (Dialogue Breakdown Detection Challenge): 200 human utterances filtered for clarity.
  • Twitter & OpenSubtitles: Subsets of 200 query-response pairs for compatibility with existing literature.
  • NCM Prompts: Prompts from the original Google Neural Conversational Model (Vinyals and Le, 2015) to enable direct comparison with this influential work.

Key Insights and Results

The research highlights several critical findings for the field:

  • The Baseline Problem: The authors found that replicating the original performance of the Google NCM model was nearly impossible, suggesting that many "state-of-the-art" claims may be measured against underperforming baselines.
  • Prompt Quality: IRT analysis revealed that open-ended prompts (e.g., "Tell me something about your parents") are far more discriminative than simple factual ones.
  • Cost-Effectiveness: A robust human evaluation (600 ratings for 200 prompts) can be conducted for approximately $7.20, making standardized human evaluation accessible for most research labs.
  • Agreement Variance: Inter-annotator agreement (Cohen’s Kappa) ranged from 0.2 to 0.54, emphasizing the need for multiple evaluators per prompt to mitigate noise in human judgment.

Limitations and Future Directions

Currently focused on English-language chatbots and static response files, the ChatEval team plans to expand the framework. Future work includes increasing dataset sizes for more stable metrics and extending the "Eval" framework to other domains like Information Retrieval (IREval), Task-based bots (TaskEval), and Natural Language Generation (NLGEval). They also aim to support real-time interaction with models via APIs.

Summary updated

BibTeX
@inproceedings{sedoc2019chateval,
  author = {João Sedoc and Daphne Ippolito and Arun Kirubarajan and Jai Thirani and Lyle Ungar and Chris Callison-Burch},
  title = {{ChatEval: A Tool for Chatbot Evaluation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rJbdpQ-OZS},
}
Figure