BatchEval: Towards Human-like Text Evaluation

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, Kan Li

Abstract

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.

Introduction

Evaluating the quality of AI-generated text—spanning metrics like coherence, factuality, and naturalness—is a cornerstone of natural language processing. While human evaluation remains the gold standard, its high cost and lack of scalability have led researchers to adopt Large Language Models (LLMs) like GPT-4 as automated judges.

However, the prevailing "sample-wise" evaluation paradigm, where an LLM assesses one text at a time, is fundamentally limited. Unlike human judges who naturally compare and contrast different examples to calibrate their scores, sample-wise LLM evaluators operate in a vacuum. This isolation leads to three critical failures: high sensitivity to prompt wording, poor resistance to textual noise (such as typos), and a lack of discriminative power, often resulting in "safe" but uninformative integer scores.

The BatchEval Paradigm

To address these limitations, we introduce BatchEval, a novel evaluation framework that mimics human comparative judgment. Instead of scoring samples in isolation, BatchEval provides the LLM with a batch of samples simultaneously, allowing the model to use inter-sample comparison as a secondary reference alongside the provided rubric.

Key Methodological Components

BatchEval's effectiveness stems from four specific design choices:

  1. Two-Stage Evaluation: The process begins with a horizontal analysis phase where the LLM examines all samples in a batch to identify relative strengths and weaknesses. Only after this comparative analysis does the model proceed to the scoring phase.
  2. Heterogeneous Batching: Rather than selecting samples randomly, BatchEval employs an iterative strategy to group samples with diverse quality levels. By presenting a spectrum of "high-quality" and "low-quality" texts together, the model can better anchor its internal scale.
  3. Decimal Scoring: While traditional evaluators often rely on coarse integers (e.g., 1-5), BatchEval encourages the use of decimals (e.g., 3.4 vs. 3.7). This allows the model to express the fine-grained distinctions it discovers during the comparison phase.
  4. Iterative Ensembling: To ensure stability, the process is repeated $N$ times with varying batch compositions. The final score is an average of these iterations, leveraging the diversity of contexts to reach a more robust conclusion.

Experimental Results

We validated BatchEval across four diverse benchmarks: Topical-Chat (dialogue), FED (multi-criteria dialogue quality), HANNA (story generation), and QAGS (factual consistency/hallucination detection).

Performance and Efficiency

BatchEval demonstrated a significant performance leap over existing state-of-the-art (SOTA) methods like G-Eval. On average, BatchEval improved Pearson correlation with human judgments by 10.5%. Remarkably, this accuracy boost was achieved with higher efficiency; BatchEval required only 64% of the API costs compared to top-performing sample-wise baselines. This efficiency is due to the token-saving nature of batch processing compared to the high-iteration counts required for stable sample-wise ensembling.

Robustness to Noise

One of the most striking findings was BatchEval's resilience. When subjected to adversarial noise (such as typos or synonym swaps), the performance of leading sample-wise methods like CloserLook plummeted by 0.109. In contrast, BatchEval’s performance remained nearly stagnant, dropping by a negligible 0.003. This suggests that the comparative context allows the model to look past surface-level errors that might otherwise bias a single-sample score.

Theoretical and Internal Analysis

The paper provides mathematical grounding for these results. Theorem 1 demonstrates that evaluators producing a more uniform score distribution—facilitated by BatchEval’s decimal scoring and batching—are inherently more resistant to noise. Furthermore, Theorem 2 suggests a "variance paradox": while we usually seek consensus in ensembling, higher variance between individual batch iterations actually contributes to a more accurate final average by capturing a wider range of comparative contexts.

Internal attention analysis confirms this mechanism. By examining the LLM's attention matrices, we observed that during the scoring phase, the model actively "looks back" at other samples within the batch to finalize its decision, confirming that the model is indeed utilizing the comparative information provided.

Limitations and Future Outlook

While BatchEval represents a significant step forward, it is subject to the constraints of current LLM architectures. The most notable limitation is the context window; if batch sizes exceed 10 samples, the model's performance begins to degrade as it becomes overwhelmed by information. Additionally, the strategy requires high-reasoning models; while GPT-4 excelled, smaller models like Llama-2-13b struggled to maintain the necessary logic for complex batch comparisons.

Future research will explore more sophisticated batch composition strategies, such as active learning, to identify the most informative sample groupings to further refine the calibration process.

Summary updated

Abstract

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.
BibTeX
@inproceedings{yuan2024batcheval,
  author = {Peiwen Yuan and Shaoxiong Feng and Yiwei Li and Xinglin Wang and Boyuan Pan and Heda Wang and Kan Li},
  title = {{BatchEval: Towards Human-like Text Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=OJrCP6PKR0},
}
Figure