A Closer Look into Using Large Language Models for Automatic Evaluation

Cheng-Han Chiang, Hung-yi LeeEMNLP 2023 · OpenReview

Abstract

Using large language models (LLMs) to evaluate text quality has recently gained popularity. While various frameworks exist, they differ in the specifics of the evaluation process. This paper analyzes LLM evaluation and G-Eval to determine how specific design choices—such as Chain-of-Thought (CoT) and output constraints—impact the correlation between LLM ratings and human judgment. We find that auto-CoT is not always beneficial and that forcing numeric-only outputs is suboptimal. Conversely, requiring the LLM to explain its ratings consistently improves performance, allowing ChatGPT to surpass state-of-the-art (SoTA) GPT-4 correlations on key benchmarks.

Introduction

As LLMs have advanced, they are increasingly used as "automatic judges" to assess generated text quality across dimensions like fluency or coherence. However, existing frameworks like G-Eval employ different "recipes": some utilize Chain-of-Thought (CoT) reasoning, while others restrict outputs to single numeric scores. The impact of these design choices on reliability has remained unclear. This study systematically deconstructs these methods to identify the most robust prompting strategies for turning LLMs into reliable evaluators.

Methodology

The researchers conducted ablation studies using ChatGPT (gpt-3.5-turbo-0613) to isolate the prompt components that drive alignment with human ratings. They compared five primary strategies:

  1. Score Only: The model outputs only a numeric value (the G-Eval baseline).
  2. Free Text: The model answers a quality-related question without format restrictions.
  3. Rate-Explain: The model provides a numeric rating first, followed by a justification.
  4. Analyze-Rate: The model analyzes the text based on specific criteria before providing a rating (zero-shot CoT).
  5. Auto-CoT: The model uses LLM-generated "evaluation steps" within the final prompt.

Performance was measured using Pearson’s $r$ and Kendall’s $\tau$ correlation coefficients against human scores on two benchmarks: SummEval (summarization) and Topical-Chat (knowledge-grounded dialogue).

Results: Summarization (SummEval)

The study revealed that sophisticated prompting can enable smaller models to outperform larger ones. Specifically, using the Analyze-rate strategy, ChatGPT achieved a Pearson correlation of 0.635 on SummEval Coherence, surpassing the previous SoTA of 0.581 set by GPT-4.

Ablation results on the SummEval dataset. The table compares GPT-4 baselines with various ChatGPT prompting strategies across Coherence, Consistency, Fluency, and Relevance.

Interestingly, the "Auto-CoT" technique used in G-Eval—where the model generates its own evaluation steps—was found to be inconsistent. In several instances, it actually decreased correlation with human scores, likely because the generated steps were redundant paraphrases of the original instructions.

Results: Dialogue (Topical-Chat)

The benefits of explanation-based prompting were even more pronounced in the dialogue domain. On the Topical-Chat dataset, transitioning from a "Score only" prompt to an Analyze-rate prompt resulted in a massive jump in Pearson correlation for "Groundedness," from 0.358 to 0.725.

Ablation results on the Topical-Chat dataset. The results demonstrate that asking for explanations (Analyze-rate) significantly boosts correlation across all metrics, especially Groundedness.

Key Insights

The "Explanation Effect"

The most significant finding is that rationalization improves correlation. Even in the Rate-explain setup—where the model provides the score before the explanation—performance improves. Since LLMs are causal and cannot "see" future text, this suggests that the instruction to explain nudges the model toward selecting a more defensible, human-aligned score from the start.

Suboptimality of Numeric Constraints

Forcing a model to output a "score only" (e.g., just the digit "5") consistently yielded worse results than allowing the model to speak freely or provide analysis. Furthermore, methods involving explanation (Analyze-rate and Rate-explain) proved much more robust to changes in sampling temperature and "persona" prompts compared to simple scoring methods.

Limitations and Future Work

While promising, the study focused primarily on ChatGPT. Preliminary tests with open-source models like Falcon-40B suggested they struggled to follow complex evaluation instructions. Additionally, the findings are currently limited to summarization and dialogue tasks; their generalizability to other NLP domains remains to be seen. The authors recommend that users verify LLM evaluator effectiveness on a small subset of their specific data due to the models' sensitivity to exact wording.

Summary updated

Abstract

Using large language models (LLMs) to evaluate text quality has recently gained popularity. Some existing prior works explore the idea of using LLMs for evaluation, while they differ in some details of the evaluation process. In this paper, we analyze *LLM evaluation* and *G-Eval*, and we discuss how those details in the evaluation process change how well the ratings given by LLMs correlate with human ratings. We find that the auto Chain-of-Thought (CoT) used in G-Eval does not always make G-Eval more aligned with human ratings. We also show that forcing the LLM to output only a numeric rating, as in G-Eval, is suboptimal. Last, we reveal that asking the LLM to explain its own ratings consistently improves the correlation between the ChatGPT and human ratings and pushes state-of-the-art (SoTA) correlations on two meta-evaluation datasets.
BibTeX
@inproceedings{chiang2023closer,
  author = {Cheng-Han Chiang and Hung-yi Lee},
  title = {{A Closer Look into Using Large Language Models for Automatic Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=RsK483IRuO},
}
Figure