Learning an Unreferenced Metric for Online Dialogue Evaluation

Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, Joelle Pineau

Abstract

Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue. There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.

Introduction

Evaluating open-domain chatbots is notoriously difficult because a single prompt can have infinite valid responses. Traditional metrics like BLEU, widely used in machine translation, fail in dialogue evaluation due to the "one-to-many" problem: a perfectly valid response might have zero word overlap with a single human reference. Furthermore, most existing metrics are "referenced," requiring a pre-written human response for comparison. This limits their utility for online evaluation, where a bot interacts with a human in real-time without a "correct" answer key.

To address these challenges, we introduce MAUDE (Metric for automatic Unreferenced dialogue evaluation). MAUDE is a self-supervised metric designed to judge if a response is appropriate, engaging, and linguistically sound based solely on the conversation's context, without needing a "gold standard" reference.

Methods

MAUDE combines the power of pre-trained transformer models with an explicit understanding of dialogue structure. The architecture consists of three main components:

  1. Utterance Encoding: The model uses DistilBERT to extract latent representations of individual sentences.
  2. Temporal Modeling: Unlike previous metrics that treat a conversation as a static bag of words, MAUDE employs a Bidirectional LSTM to model the transitions between utterances. This allows the model to assess whether a sentence logically and stylistically follows the preceding context.
  3. Noise Contrastive Estimation (NCE): To train without human-labeled scores, MAUDE uses a contrastive approach. It learns to distinguish real dialogue pairs (positive samples) from "corrupted" versions (negative samples). Corruptions include syntactic noise (word shuffling, dropping, or repeating) and semantic noise (swapping in random sentences or using back-translation paraphrases).
The MAUDE architecture for unreferenced dialogue evaluation.

Experimental Setup

The model was primarily trained and analyzed using the PersonaChat dataset. To test generalization, the authors evaluated it on DailyDialog (human-centric chit-chat) and goal-oriented datasets like Frames and MultiWOZ. The evaluation focused on how well the metric correlates with human judgments and its robustness against adversarial inputs.

Results

MAUDE demonstrates significant improvements over baseline metrics such as RUBER and InferSent.

Human Correlation

MAUDE achieved higher Spearman correlation with human judgments across most categories, showing particular strength in measuring Interestingness and Engagingness. By leveraging pre-trained DistilBERT embeddings, it maintains high performance even on datasets it has never seen before (zero-shot generalization).

Correlation of MAUDE and baseline metrics with human scores across various dialogue quality dimensions.

Robustness and Generalization

In adversarial tests, MAUDE was significantly better at penalizing poor responses, such as word repetitions or irrelevant injections, compared to BERT-NLI baselines. When tested on DailyDialog after training on PersonaChat, MAUDE showed a higher "Delta" score—the gap between its rating of a good versus a bad response—proving its robustness beyond its training distribution.

Discussion and Conclusion

The research highlights that dialogue has a "rhythm"; BERT embeddings cluster based on an utterance's position in a conversation. Capturing these temporal transitions via an RNN is crucial for performance. Furthermore, a balanced training diet of both syntactic and semantic noise is essential; training only on syntactic noise leaves a model blind to semantic irrelevance.

Despite its strengths, MAUDE shares a "repetition blind spot" with other BERT-based models, sometimes failing to penalize a bot that repeats a high-quality sentence. Future work involves using MAUDE as a reward function in Reinforcement Learning to train dialogue models that are more naturally engaging and less prone to generic responses.

Summary updated

Abstract

Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue. There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
BibTeX
@inproceedings{sinha2020learning,
  author = {Koustuv Sinha and Prasanna Parthasarathi and Jasmine Wang and Ryan Lowe and William L. Hamilton and Joelle Pineau},
  title = {{Learning an Unreferenced Metric for Online Dialogue Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.220},
}
Figure