Abstract
Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigms suffer from prompt sensitivity, poor noise resistance, and inferior ensemble performance. Inspired by human evaluation processes that utilize both criterion definitions and inter-sample comparisons, we propose BatchEval. This paradigm conducts iterative, batch-wise evaluation to alleviate these issues. Our findings indicate that a two-stage procedure with heterogeneous batch composition and decimal scoring is optimal. Experiments across three LLMs and four evaluation tasks show that BatchEval outperforms state-of-the-art methods by 10.5% in Pearson correlation while reducing API costs by 36% on average.
Introduction: The Limits of Sample-wise Evaluation
Evaluating the quality of AI-generated text—spanning dimensions like coherence, naturalness, and engagingness—remains a significant challenge. While the field has transitioned from simple n-gram overlap metrics (e.g., BLEU) to LLM-based judges (e.g., G-Eval), these models typically operate under a sample-wise paradigm. In this setup, the LLM evaluates a single response against a static rubric in isolation.
This "vacuum-sealed" approach has three major flaws:
- Prompt Sensitivity: Minor variations in rubric phrasing can lead to significant score fluctuations.
- Lack of Discrimination: Without external reference points, LLMs often suffer from "score clumping" (e.g., assigning a 3/5 to most samples), reducing robustness against text noise.
- Static Ensembles: Repeatedly scoring the same sample provides insufficient diversity for effective ensemble averaging.
In contrast, human evaluators do not rely solely on rubrics; they calibrate their internal scales by comparing samples against one another.
Methodology: The BatchEval Framework
BatchEval replicates this human-like comparative process through an iterative, batch-wise workflow. Instead of processing samples individually, the framework organizes them into batches and follows a structured evaluation pipeline.
Key Components
- Heterogeneous Batching: Samples are grouped into batches that mix previously high-scoring and low-scoring examples. These "anchors" provide the LLM with clear references for the quality spectrum.
- Two-Stage Procedure: The LLM first performs a horizontal analysis of all samples in the batch (Stage 1) before assigning final scores based on that comparative context (Stage 2).
- Decimal Scoring: By moving away from coarse integers (1-5) to decimal formats (e.g., 2.2 vs 1.8), BatchEval allows the model to express the subtle distinctions identified during the comparison stage.
- Iterative Refinement: The process runs for $N$ rounds. Batch compositions are dynamically optimized based on scores from previous rounds to ensure a wide variety of reference points.
Experimental Setup and Results
The researchers validated BatchEval across four diverse benchmarks: Topical-Chat (dialogue), FED (dialogue-level likeability), HANNA (story generation), and QAGS (summarization factual consistency).
Performance and Efficiency
- Superior Accuracy: BatchEval achieved a 10.5% improvement in Pearson correlation with human judgments compared to SOTA baselines like G-Eval and CloserLook.
- Cost Efficiency: By processing multiple samples within a single prompt, BatchEval reduced API costs to just 64% of the leading baselines.
- Robustness: The method demonstrated lower variance when prompts were rewritten and maintained performance even when input text was corrupted with noise.
Discussion and Insights
The core innovation of BatchEval is treating in-batch samples as an "orthogonal" reference to the evaluation criteria. This reduces over-reliance on specific prompt wording. Furthermore, the authors demonstrated that a more uniform score distribution—a natural byproduct of relative ranking—enhances robustness.
Attention matrix visualizations using Llama-2-70b confirmed the mechanism: during the scoring phase, the model actively "looks back" at the analyses of other samples in the batch to calibrate the final output.
Limitations
While effective, BatchEval is constrained by the context windows of current LLMs. Additionally, while it excels at stylistic dimensions like "Naturalness," the gains were more modest for tasks requiring deep logical reasoning, such as "Factual Consistency." Future work may explore adaptive batching and more complex ranking-then-scoring workflows.
Summary updated