ChatEval: A Tool for the Systematic Evaluation of Chatbots

João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, Chris Callison-Burch2018 · DOI

Abstract

Open-domain dialog systems are difficult to evaluate. The current best practice for analyzing and comparing these dialog systems is the use of human judgments. However, the lack of standardization in evaluation procedures, and the fact that model parameters and code are rarely published hinder systematic human evaluation experiments. We introduce a unified framework for human evaluation of chatbots that augments existing chatbot tools, and provides a web-based hub for researchers to share and compare their dialog systems. Researchers can submit their trained models to the ChatEval web interface and obtain comparisons with baselines and prior work. The evaluation code is open-source to ensure evaluation is performed in a standardized and transparent way. In addition, we introduce open-source baseline models and evaluation datasets. ChatEval can be found at https://chateval.org.

Introduction

Evaluating open-domain chatbots is notoriously difficult. While human judgment is considered the gold standard, the process is currently fragmented and plagued by a lack of reproducibility. Researchers often use different prompt sets and evaluation scales, and rarely release their trained model parameters or code. This lack of standardization leads to a "silo" effect: instead of comparing a new model against the current state-of-the-art, researchers often just compare it against a basic sequence-to-sequence (Seq2Seq) baseline.

ChatEval aims to provide the "missing infrastructure" to make these comparisons transparent and consistent. It is a unified, web-based framework designed to standardize how we measure open-domain dialog systems, providing a central hub where researchers can submit model responses to be benchmarked using both automated metrics and human A/B testing against established baselines.

The ChatEval Framework

The authors built ChatEval as a two-part system: a web portal for the community and an open-source evaluation toolkit.

The ChatEval workflow for model submission and evaluation.
  1. The Web Interface: Researchers submit their model’s responses to a standard set of prompts. The site then hosts a Model Profile (showing responses and metrics) and a Response Comparison tool that allows for side-by-side qualitative analysis of how different models handle the same input.
  2. Automatic Evaluation: The toolkit computes several standard metrics, including diversity (unique n-grams), semantic similarity (cosine similarity of word embeddings), BLEU-2 scores, and perplexity.
  3. Human Evaluation: This is the core of the framework. ChatEval facilitates A/B comparison tests via Amazon Mechanical Turk (AMT). Instead of asking workers to give an absolute score (e.g., "rate this 1-5"), which is often inconsistent, workers are shown a prompt and two different model responses and asked to pick the better one or declare a tie.
The human evaluation interface used for A/B testing on Amazon Mechanical Turk.

Standardized Datasets

ChatEval introduces and curates several evaluation sets to ensure models are tested on diverse conversational styles:

  • DBDC (Dialogue Breakdown Detection Challenge): 200 human utterances filtered for clarity, with new human reference responses collected by the authors.
  • Twitter & OpenSubtitles: Subsets of 200 query-response pairs from these common datasets.
  • NCM Prompts: The specific 200 prompts used in the original 2015 Neural Conversational Model paper, allowing for direct historical comparison.

Key Insights and Results

A striking finding was the "reproducibility gap": no publicly available version of the NCM (including those trained by the authors) matched the performance reported in the original paper. This underscores why a central repository for trained parameters—not just code—is vital.

The authors argue that direct A/B comparisons are much more discriminative than absolute ratings. In absolute rating tasks, most models tend to cluster within one standard deviation of each other, making it hard to distinguish quality. By using three evaluators for 200 prompt/response pairs, the authors established a baseline cost for human evaluation of approximately $6.

Inter-Annotator Agreement (IAA), measured using Cohen’s weighted kappa, typically ranges between 0.2 and 0.54. This variation highlights how subjective dialog evaluation remains, even with a standardized framework. The launch of chateval.org provides a living leaderboard where results are posted publicly.

Conclusion and Future Work

While the current toolkit supports several automatic metrics, the system is designed to be extensible. Future work will involve incorporating newer, more sophisticated "learned" evaluation metrics as the field moves away from simple n-gram overlaps. The success of the platform depends on community adoption—researchers must be willing to share their model outputs to grow the ecosystem and bridge the gap in chatbot evaluation.

Summary updated

Abstract

Open-domain dialog systems are difficult to evaluate. The current best practice for analyzing and comparing these dialog systems is the use of human judgments. However, the lack of standardization in evaluation procedures, and the fact that model parameters and code are rarely published hinder systematic human evaluation experiments. We introduce a unified framework for human evaluation of chatbots that augments existing chatbot tools, and provides a web-based hub for researchers to share and compare their dialog systems. Researchers can submit their trained models to the ChatEval web interface and obtain comparisons with baselines and prior work. The evaluation code is open-source to ensure evaluation is performed in a standardized and transparent way. In addition, we introduce open-source baseline models and evaluation datasets. ChatEval can be found at https: //chateval.org.
BibTeX
@misc{sedoc2018chateval,
  author = {João Sedoc and Daphne Ippolito and Arun Kirubarajan and Jai Thirani and Lyle Ungar and Chris Callison-Burch},
  title = {{ChatEval: A Tool for the Systematic Evaluation of Chatbots}},
  year = {2018},
  doi = {10.18653/v1/w18-6709},
}
Figure