TL;DR
The NTCIR-18 AEOLLM task addresses the critical need for better automatic evaluation of Large Language Models (LLMs) by focusing on generative tasks and encouraging reference-free methods. This initiative introduces a diverse set of challenging subtasks and a new dataset to push the boundaries of LLM evaluation beyond traditional, limited approaches.
What Problem Does This Paper Address?
As LLMs like GPT-4 and Qwen become increasingly powerful and ubiquitous, evaluating their capabilities effectively is a major bottleneck. Current automatic evaluation methods often fall short: they are frequently limited to multiple-choice questions, which don't reflect real-world generative tasks, and rely heavily on reference-based metrics (like ROUGE or BLEU). These reference-based metrics are problematic because LLMs can easily "memorize" and optimize for them during training, rendering them less effective for truly assessing novel performance. Furthermore, manual evaluation, while more robust, is prohibitively expensive and time-consuming. The AEOLLM task aims to bridge this gap by developing more sophisticated, reference-free automatic evaluation methods for LLMs on generative tasks.
Methods
The AEOLLM task is designed to encourage innovation in automatic LLM evaluation. Key aspects of its methodology include:
- Focus on Generative Tasks: Unlike many existing benchmarks, AEOLLM centers on tasks that require LLMs to generate novel content, such as dialogue, text expansion, summarization, and non-factoid question answering.
- Encouraging Reference-Free Methods: The task explicitly promotes the development of evaluation methods that do not rely on pre-defined reference answers. This is crucial for overcoming the "memorization" issue with LLMs.
- Diverse Subtasks: To comprehensively test LLM evaluation capabilities, the task includes four distinct subtasks:
- Dialogue Generation: Generating human-like responses in conversational contexts.
- Text Expansion: Creating stories based on given prompts.
- Summary Generation: Producing concise summaries of longer texts.
- Non-Factual Question Answering: Answering questions that don't have a single correct answer.
- Human Annotation as Gold Standard: While promoting automatic evaluation, the task uses human annotators to provide scores and rankings for LLM-generated answers. These human judgments serve as the "gold standard" against which automatic evaluation methods are compared.
- Evaluation Metrics: The task employs a combination of metrics to assess the consistency between automatic evaluations and human judgments:
- Accuracy (acc): Measures the proportion of identical preference results between the model and human annotations.
- Kendall's Tau (τ): Assesses the ordinal association between two ranked variables.
- Spearman's Rank Correlation Coefficient (ρ): Measures the strength and direction of association between two ranked variables.
Datasets
The AEOLLM task utilizes existing datasets adapted for its specific evaluation needs:
- Dialogue Generation: DailyDialog [12]
- Text Expansion: Writingprompts [6]
- Summary Generation: Extreme Summary (Xsum) [16]
- Non-Factual Question Answering: NF_CATS [3]
For each dataset, 100 instances were sampled to form the question set. Seven different LLMs were used to generate answers, creating an answer set. This combined set of 2,800 instances was then split into:
- Training Set (20%): Included human annotations for participants to develop their methods.
- Test Set (20%): Used for evaluating participant methods and generating leaderboards.
- Reserved Set (60%): Used for the final evaluation and reporting of official results.
Key Insights and Ideas
- The Need for Diverse Evaluation: The results show that different LLMs excel on different subtasks and that different evaluation metrics (accuracy vs. Kendall's Tau/Spearman's rho) capture different aspects of performance. This highlights that a single metric or a single task is insufficient for a comprehensive LLM evaluation.
- Text Expansion is Particularly Challenging: The Text Expansion dataset proved to be the most difficult for evaluation, with the lowest accuracy scores. This is attributed to the longer answer lengths in story generation, making it harder for evaluation methods to accurately assess quality.
- LLMs as Judges Show Promise: A simple baseline using GPT-4o directly for evaluation achieved strong results on Text Expansion and Non-Factual QA. This suggests that current powerful LLMs can already perform well as evaluators ("LLMs-as-judges"), offering a promising direction for future research.
- Reference-Free is Key: The task's emphasis on reference-free methods is crucial for moving beyond superficial performance metrics and truly assessing LLM capabilities in real-world generative scenarios.
Key Results
The NTCIR-18 AEOLLM task received 48 runs from 4 teams.
- Overall Performance: PanguIR achieved the best overall accuracy (acc), while UCLWI led in Kendall's Tau (τ) and Spearman's Rank correlation coefficients (ρ).
- Subtask Performance:
- UCLWI excelled in Dialogue Generation (all metrics) and in τ and ρ for Story Generation.
- PanguIR outperformed others in acc for Text Expansion and Non-Factual QA.
- ISLab led in acc for Summary Generation.
- Challenging Subtask: Text Expansion had the highest acc of only 0.5581, indicating its difficulty.
- Simple Baseline Success: A baseline using GPT-4o directly achieved the best τ and ρ in Text Expansion and Non-Factual QA, demonstrating the potential of LLMs as evaluators.
Limitations and Future Work
- Dataset Difficulty: The Text Expansion dataset is identified as particularly challenging for evaluation, presenting an opportunity for future method optimization.
- Metric Complementarity: The paper emphasizes that using multiple metrics is necessary for a comprehensive assessment, as accuracy can sometimes differ from rank correlation coefficients.
- Future Directions: The authors plan to further extend the AEOLLM task to "better and more comprehensively evaluate LLMs," suggesting ongoing development and refinement of the task and its associated datasets and methodologies.
Summary updated