Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric

Ian Berlot-Attwell, Frank Rudzicz

TL;DR

Existing dialogue relevance metrics often perform poorly and are highly sensitive to the specific dataset they're evaluated on. This paper introduces a new, simpler metric called "IDK" that achieves state-of-the-art performance on one dataset and demonstrates significantly reduced domain sensitivity, all while requiring minimal training data and no fine-tuning of large language models.

What Problem Does This Paper Address?

Evaluating the quality of conversational AI, particularly dialogue systems, is a major challenge. One crucial aspect of this evaluation is "relevance," which essentially asks if a response is on-topic and makes sense in the context of the ongoing conversation. While many automatic metrics have been proposed to measure this, their effectiveness is often limited. A key problem is that these metrics tend to be highly dependent on the specific dataset they were trained or evaluated on, making it difficult to trust their performance on new, unseen data or different dialogue domains. This lack of generalization means that a metric that works well for one type of chatbot might fail miserably for another. Furthermore, many existing metrics require expensive human annotations or fine-tuning large pre-trained models, which can be impractical. The authors aim to address this by developing a relevance metric that is more robust, less data-hungry, and generalizes better across different dialogue contexts.

Methods

The core of the paper revolves around evaluating existing relevance metrics and proposing a new one, "IDK" (I Don't Know).

Evaluation of Existing Metrics: The authors first systematically evaluated six existing dialogue relevance metrics on four different datasets. These metrics included:

  • Cosine Similarity based: COS-FT (using fastText embeddings) and COS-MAX-BERT/COS-NSP-BERT (using BERT embeddings).
  • Next Utterance Prediction (NUP) based: NUP-BERT, which fine-tunes a BERT model for this task.
  • Normalized Conditional Probability based: NORM-PROB (using GPT-2) and FED-RELEVANT/FED-CORRECT (using DialoGPT).
  • Dialogue Coherence Metrics (used for comparison): GRADE and DYNA-EVAL.

Proposed Metric (IDK): The IDK metric builds upon the NUP-BERT architecture but introduces several key modifications:

  • Frozen BERT Features: Instead of fine-tuning BERT, IDK uses pre-trained BERT Next Sentence Prediction (NSP) features directly. This significantly reduces data requirements.
  • Single Fixed Negative Example: Instead of using randomly sampled negative responses, IDK uses a single, fixed negative example: "i don't know." This drastically simplifies training and data needs.
  • L1 Regularization: To further improve generalization and handle the single negative example, L1 regularization is applied to the logistic regression weights.
  • Logistic Regression: A simpler logistic regression classifier is used instead of a two-class softmax.

Datasets

The study utilized four publicly available English dialogue datasets, all annotated with human relevance ratings:

  • HUMOD Dataset: A subset of the Cornell movie dialogue dataset, containing 4,750 contexts with human-annotated relevance scores (Likert 1-5).
  • USR Topical-Chat Dataset (USR-TC): A subset of Topical-Chat, with 60 contexts and human relevance scores (Likert 1-3).
  • Pang et al. (2020) Annotated DailyDialogue Dataset (P-DD): A subset of DailyDialogue, with 200 contexts and ten crowdsourced relevance ratings (Likert 1-5).
  • FED Dataset: Consists of 375 annotated dialogue turns from human-human, human-Meena, and human-Mitsuku conversations, evaluated for both relevance and correctness.

Key Insights and Ideas

  • Existing metrics are brittle: The study strongly suggests that most current dialogue relevance metrics are highly sensitive to the dataset and often generalize poorly, sometimes even showing negative correlation with human judgments on new datasets.
  • Less is more (sometimes): The IDK metric demonstrates that using a single, fixed negative example ("i don't know") can be as effective, if not more so, than random negative sampling for training a relevance metric, especially when combined with L1 regularization.
  • BERT NSP features are surprisingly robust: The research found that a vast majority (over 99%) of BERT NSP features can be discarded without significant impact on performance, suggesting that only a few key dimensions capture most of the relevant information for this task.
  • Domain sensitivity is a major issue: The authors quantify domain sensitivity and show that their proposed IDK metric significantly outperforms existing methods in this regard, indicating greater robustness across different dialogue types.
  • Minimal data can be sufficient: IDK's ability to perform well with only 3,750 unannotated dialogues and a single negative example highlights a path towards more accessible relevance evaluation.

Key Results

  • State-of-the-art on HUMOD: The proposed IDK metric achieved state-of-the-art Spearman correlation on the HUMOD dataset, surpassing previous benchmarks.
  • Reduced Domain Sensitivity: IDK reduced domain sensitivity by 37%-66% compared to the best baseline, meaning its performance is more consistent across different datasets.
  • Competitive Performance: Despite its simplicity and minimal data requirements, IDK demonstrated competitive performance across four diverse datasets.
  • NUP-BERT Limitations: Even fine-tuned NUP-BERT showed significant limitations, with weak performance on HUMOD and FED, and sensitivity to the training domain.
  • Cosine and Probability Metrics Struggle: Metrics based on cosine similarity and normalized probability generally showed poor generalization and were often negatively correlated with human ratings on unseen datasets.

Limitations and Future Work

  • Model Evaluation vs. Prototyping: The authors acknowledge that while IDK is useful for rapid prototyping and hyperparameter search, it might not be suitable for final model evaluation due to its reliance on BERT NSP features, which could unfairly favor models built on similar architectures.
  • Understanding "Why": The paper raises questions about why certain metrics fail to generalize and why discarding most BERT NSP features works. Further theoretical investigation is needed.
  • Counter-intuitive findings: The success of a single negative example and the minimal impact of discarding BERT features are counter-intuitive and warrant deeper exploration.
  • Further exploration of negative samples: While "i don't know" was effective, exploring other fixed negative samples could yield further insights, though it might introduce more domain sensitivity.
  • Triplet Loss: Experiments with triplet loss showed mixed results, sometimes leading to degenerate solutions, suggesting it's not a clear improvement over BCE loss for this specific setup.

Summary updated

Abstract

In this work, we evaluate various existing dialogue relevance metrics, find strong dependency on the dataset, often with poor correlation with human scores of relevance, and propose modifications to reduce data requirements and domain sensitivity while improving correlation. Our proposed metric achieves state-of-the-art performance on the HUMOD dataset while reducing measured sensitivity to dataset by 37%-66%. We achieve this without fine-tuning a pretrained language model, and using only 3,750 unannotated human dialogues and a single negative example. Despite these limitations, we demonstrate competitive performance on four datasets from different domains. Our code, including our metric and experiments, is open sourced.
BibTeX
@inproceedings{berlotattwell2022relevance,
  author = {Ian Berlot-Attwell and Frank Rudzicz},
  title = {{Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric}},
  booktitle = {ConvAI @ ACL},
  year = {2022},
  doi = {10.18653/v1/2022.nlp4convai-1.14},
}
Figure