Abstract
Evaluating the relevance of responses in generative dialogue systems remains a significant challenge. Existing metrics often fail to generalize across diverse datasets, showing inconsistent or even negative correlations with human judgment when applied to new domains. This paper introduces IDK, a minimalist relevance metric that utilizes frozen BERT features and a single fixed negative example ("i don't know."). Our empirical evaluation across four diverse datasets demonstrates that IDK achieves state-of-the-art performance and superior domain robustness compared to more complex, fine-tuned models.
Introduction
The ability of a chatbot to provide relevant, on-topic responses is fundamental to effective human-AI interaction. However, measuring "relevance" is notoriously difficult. While metrics for fluency and engagement exist, relevance metrics often suffer from fragility. Most are developed on narrow datasets and fail to maintain performance when transferred to different conversational contexts. This "domain sensitivity" necessitates a more robust, "plug-and-play" approach to dialogue evaluation.
Benchmarking Existing Metrics
The authors first evaluated several established approaches, including n-gram methods, embedding similarities (e.g., COS-FT, COS-MAX-BERT), and fine-tuned models like NUP-BERT (Next Utterance Prediction). The results highlighted a sobering reality: many popular metrics are heavily overfitted. For instance, a metric performing well on one dataset might show near-zero or negative correlation on another. This inconsistency underscores the need for a metric that captures the underlying semantic boundary of relevance rather than dataset-specific artifacts.
The IDK Metric: A Minimalist Approach
To address these limitations, the authors propose IDK ("I Don't Know"). The architecture is intentionally simple, designed to avoid overfitting and maximize generalization:
- Frozen BERT Features: Instead of computationally expensive fine-tuning, IDK extracts pre-trained Next Sentence Prediction (NSP) features from a standard BERT model.
- Logistic Regression with L1 Regularization: A simple logistic regression classifier is trained on these features. Strong L1 regularization is applied to identify the most critical dimensions for relevance.
- The "IDK" Negative Sampling: Unlike standard "random negative sampling," which uses random sentences as negative examples, IDK uses the fixed phrase "i don't know." as the sole negative example during training. This prevents the model from learning trivial syntactic differences and forces it to focus on the semantic transition between a context and a response.
Experimental Setup
The metrics were evaluated across four diverse English dialogue datasets, each containing human Likert-scale annotations for relevance:
- HUMOD: Derived from the Cornell Movie Dialogue dataset.
- USR-TC: Based on the Topical-Chat dataset.
- P-DD: A subset of DailyDialogue with synthetic responses.
- FED: Includes both human-human and human-bot (Meena, Mitsuku) interactions.
Key Results and Insights
Performance and Robustness
IDK achieved state-of-the-art (SOTA) Spearman and Pearson correlations with human scores, particularly on the HUMOD dataset. More importantly, it demonstrated remarkable consistency across all tested domains.
The authors quantified domain sensitivity as the ratio of the best Spearman correlation to the worst across all datasets. A lower ratio indicates higher robustness. IDK achieved a ratio of 3.9, significantly outperforming the next best baseline, NUP-BERT, which scored 6.2.
The "Less is More" Discovery
Ablation studies revealed that relevance is captured in a surprisingly small subspace of BERT's embeddings. By applying strong L1 regularization, the authors found they could zero out 761 of the 768 dimensions—leaving only 7 dimensions—without a significant loss in accuracy. This suggests that the semantic signal for dialogue relevance is highly concentrated.
Furthermore, training on a single, fixed negative example ("i don't know.") proved as effective as random sampling. This minimalist training strategy prevents the model from "cheating" by learning to distinguish between different sentence structures, focusing instead on the core concept of a relevant follow-up.
Conclusion and Future Work
IDK offers a highly efficient, robust, and "plug-and-play" solution for dialogue relevance evaluation. By utilizing frozen features and minimal training data (3,750 unannotated dialogues), it bypasses the need for expensive fine-tuning while achieving superior generalization.
While IDK is primarily intended for rapid prototyping and hyperparameter optimization, its success suggests that "less is more" in the context of relevance metrics. Future research will investigate if this minimalist approach extends to other languages and more nuanced dialogue qualities such as empathy or interestingness.
Summary updated