BatchEval: Towards Human-like Text Evaluation

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, Yao Hu, Kan Li

Abstract

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigms suffer from prompt sensitivity, poor noise resistance, and inferior ensemble performance. Inspired by human evaluation processes that utilize both criterion definitions and inter-sample comparisons, we propose BatchEval. This paradigm conducts iterative, batch-wise evaluation to alleviate these issues. Our findings indicate that a two-stage procedure with heterogeneous batch composition and decimal scoring is optimal. Experiments across three LLMs and four evaluation tasks show that BatchEval outperforms state-of-the-art methods by 10.5% in Pearson correlation while reducing API costs by 36% on average.

Introduction: The Limits of Sample-wise Evaluation

Evaluating the quality of AI-generated text—spanning dimensions like coherence, naturalness, and engagingness—remains a significant challenge. While the field has transitioned from simple n-gram overlap metrics (e.g., BLEU) to LLM-based judges (e.g., G-Eval), these models typically operate under a sample-wise paradigm. In this setup, the LLM evaluates a single response against a static rubric in isolation.

Conceptual comparison between human-like comparative evaluation and traditional sample-wise LLM evaluation.

This "vacuum-sealed" approach has three major flaws:

  1. Prompt Sensitivity: Minor variations in rubric phrasing can lead to significant score fluctuations.
  2. Lack of Discrimination: Without external reference points, LLMs often suffer from "score clumping" (e.g., assigning a 3/5 to most samples), reducing robustness against text noise.
  3. Static Ensembles: Repeatedly scoring the same sample provides insufficient diversity for effective ensemble averaging.

In contrast, human evaluators do not rely solely on rubrics; they calibrate their internal scales by comparing samples against one another.

Methodology: The BatchEval Framework

BatchEval replicates this human-like comparative process through an iterative, batch-wise workflow. Instead of processing samples individually, the framework organizes them into batches and follows a structured evaluation pipeline.

The BatchEval iterative workflow featuring heterogeneous batching and a two-stage evaluation procedure.

Key Components

  • Heterogeneous Batching: Samples are grouped into batches that mix previously high-scoring and low-scoring examples. These "anchors" provide the LLM with clear references for the quality spectrum.
  • Two-Stage Procedure: The LLM first performs a horizontal analysis of all samples in the batch (Stage 1) before assigning final scores based on that comparative context (Stage 2).
  • Decimal Scoring: By moving away from coarse integers (1-5) to decimal formats (e.g., 2.2 vs 1.8), BatchEval allows the model to express the subtle distinctions identified during the comparison stage.
  • Iterative Refinement: The process runs for $N$ rounds. Batch compositions are dynamically optimized based on scores from previous rounds to ensure a wide variety of reference points.

Experimental Setup and Results

The researchers validated BatchEval across four diverse benchmarks: Topical-Chat (dialogue), FED (dialogue-level likeability), HANNA (story generation), and QAGS (summarization factual consistency).

Performance and Efficiency

  • Superior Accuracy: BatchEval achieved a 10.5% improvement in Pearson correlation with human judgments compared to SOTA baselines like G-Eval and CloserLook.
  • Cost Efficiency: By processing multiple samples within a single prompt, BatchEval reduced API costs to just 64% of the leading baselines.
  • Robustness: The method demonstrated lower variance when prompts were rewritten and maintained performance even when input text was corrupted with noise.

Discussion and Insights

The core innovation of BatchEval is treating in-batch samples as an "orthogonal" reference to the evaluation criteria. This reduces over-reliance on specific prompt wording. Furthermore, the authors demonstrated that a more uniform score distribution—a natural byproduct of relative ranking—enhances robustness.

Attention matrix visualizations using Llama-2-70b confirmed the mechanism: during the scoring phase, the model actively "looks back" at the analyses of other samples in the batch to calibrate the final output.

Limitations

While effective, BatchEval is constrained by the context windows of current LLMs. Additionally, while it excels at stylistic dimensions like "Naturalness," the gains were more modest for tasks requiring deep logical reasoning, such as "Factual Consistency." Future work may explore adaptive batching and more complex ranking-then-scoring workflows.

Summary updated

Abstract

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.
BibTeX
@inproceedings{yuan2024batcheval,
  author = {Peiwen Yuan and Shaoxiong Feng and Yiwei Li and Xinglin Wang and Boyuan Pan and Heda Wang and Yao Hu and Kan Li},
  title = {{BatchEval: Towards Human-like Text Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=OJrCP6PKR0},
}
Figure