Abstract
Advancements in dialogue systems powered by large language models (LLMs) have outpaced the development of reliable evaluation metrics, particularly for diverse and creative responses. We present a benchmark for evaluating the robustness of reference-free dialogue metrics against four categories of adversarial attacks: speaker tag prefixes, static responses, ungrammatical responses, and repeated conversational context. We analyze metrics such as DialogRPT, UniEval, and PromptEval — a prompt-based method leveraging LLMs — across grounded and ungrounded datasets. By examining both their correlation with human judgment and susceptibility to adversarial attacks, we find that these two axes are not always aligned; metrics that appear to be equivalent when judged by traditional benchmarks may, in fact, vary in their scores of adversarial responses. These findings motivate the development of nuanced evaluation frameworks to address real-world dialogue challenges.
Introduction
Evaluating dialogue systems is a notoriously difficult challenge in natural language processing. Traditional "reference-based" metrics, such as BLEU or ROUGE, operate by comparing a model's generated response to a single human-provided "gold standard." However, in open-ended conversation, there are often thousands of valid ways to respond to a prompt. Consequently, reference-based metrics frequently penalize creative or diverse outputs that, while perfectly valid to a human, do not match the specific wording of the reference.
To address this, the field has shifted toward "reference-free" metrics. These evaluators, such as UniEval or LLM-based scorers, judge a response based solely on its relationship to the conversation history. While these metrics demonstrate a higher correlation with human intuition, their vulnerability to manipulation remains under-explored. If a metric can be easily "gamed" into assigning high scores to nonsensical or repetitive responses, it cannot be considered a reliable tool for AI development.
The Adversarial Robustness Benchmark
This research introduces a new benchmark designed to test the robustness of reference-free metrics through four distinct categories of adversarial attacks:
- Speaker Tags: Prepending labels like "teacher:" or "user:" to the response to see if the metric is biased toward specific personas.
- Static Responses: Utilizing generic, "safe" phrases (e.g., "I don't know") or grammatically correct but entirely irrelevant sentences.
- Ungrammatical Responses: Corrupting human-reference responses by removing punctuation, stripping stopwords, or jumbling word order to test if the metric can detect linguistic degradation.
- Context Repetition: Repeating the last user utterance or a specific fact from a provided knowledge base to see if the metric rewards mere mimicry.
Methodology
We evaluated three primary types of metrics across two datasets: DailyDialog (casual, ungrounded conversation) and TopicalChat (knowledge-grounded dialogue).
- DialogRPT: A specialized model trained on Reddit feedback loops.
- UniEval: A multi-dimensional evaluator based on the T5 architecture.
- PromptEval (PE): An LLM-based approach using models like GPT-3.5, GPT-4o, Llama 2, and Mixtral. We compared two scoring methods: Direct Scoring (asking the LLM for a 1-5 integer) and Weighted Scoring (calculating a score based on the log probabilities of the model's output tokens).
Key Insights
The Alignment-Robustness Gap
A central finding of this study is that alignment with human judgment does not equate to robustness. A metric may correlate well with human scores on standard datasets but fail catastrophically when faced with adversarial examples. For instance, a metric might correctly identify high-quality responses in a clean dataset but fail to penalize a response that simply repeats the user's previous sentence.
The "Teacher" Bias and Model Failures
Our results confirm that certain metrics, particularly DialogRPT, exhibit a significant bias toward speaker tags. Simply adding a "teacher:" prefix can artificially inflate scores, likely because the model interprets the tag as a proxy for informative content. Furthermore, DialogRPT showed the lowest overall correlation with human judgment, suggesting it is increasingly obsolete for modern dialogue evaluation.
Weighted Scoring as a Defense
For LLM-based evaluators, the method of extracting the score is critical. Using Weighted Scoring (averaging token probabilities) significantly improved robustness compared to Direct Scoring. This approach reduces sensitivity to prompt phrasing and provides a more granular, stable evaluation of the response quality.
Dataset and Model Performance
- The Winner: GPT-4o, when paired with weighted scoring, emerged as the most robust and human-aligned metric.
- UniEval's Efficiency: Despite having only 800 million parameters, UniEval performed remarkably well, particularly on grounded data. However, it proved vulnerable to "context repetition" attacks in casual dialogue, where it often ranked repetitive adversarial responses higher than human references.
- Grounded vs. Ungrounded: Attacks were generally more successful on the ungrounded DailyDialog dataset. The presence of external facts in TopicalChat provides a stronger "signal," making it more difficult for simple adversarial tricks to pass as high-quality dialogue.
Conclusion
This research highlights a critical "robustness gap" in current dialogue evaluation. As we move toward more sophisticated conversational agents, our evaluation tools must be able to distinguish between genuine linguistic quality and simple adversarial shortcuts. Future work should focus on expanding these benchmarks to include more complex, human-like adversarial failures and ensuring that metrics remain robust as the underlying LLMs continue to evolve.
Summary updated