MATEval: A Multi-agent Discussion Framework for Advancing Open-Ended Text Evaluation

Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, Dehai Min

Introduction

Evaluating the quality of open-ended text—such as creative stories, essays, and long-form narratives—remains one of the most significant challenges in Natural Language Processing (NLP). Traditional automated metrics like BLEU, ROUGE, and METEOR are fundamentally limited; they rely on word-overlap with reference texts, failing to account for semantic depth, logical flow, or creative nuance. While recent advancements have utilized Large Language Models (LLMs) as evaluators (e.g., G-Eval), single-agent systems often suffer from instability, high uncertainty, and a lack of transparent reasoning.

To address these limitations, this paper introduces MATEval, a multi-agent framework designed to emulate human-like collaborative discussion. By leveraging a structured interaction protocol, MATEval provides more reliable, diagnostic, and human-aligned evaluations of open-ended text.

The MATEval Framework

MATEval moves beyond the "single-judge" paradigm by employing a multi-role architecture that facilitates rigorous debate and consensus-building. The framework consists of three primary agent roles:

  1. Evaluator Agents: Multiple instances of an LLM (such as GPT-4) that independently analyze the target text.
  2. Feedback Agent: Acting as a moderator, this agent monitors the discussion rounds, identifies points of disagreement, and provides steering suggestions to help the group reach a reasoned consensus.
  3. Summarizer Agent: This agent synthesizes the entire multi-round dialogue into a final, structured diagnostic report.

Strategic Integration: SR + CoT

A core innovation of MATEval is the simultaneous application of Self-Reflection (SR) and Chain-of-Thought (CoT) strategies. While CoT allows agents to decompose complex evaluation criteria into manageable sub-questions (e.g., checking for specific logical contradictions), SR enables agents to update their perspectives based on peer input. This combination prevents "rigid thinking" and ensures that the final evaluation is grounded in a comprehensive analysis of the text's specific attributes.

Methodology and Experimental Setup

The MATEval process follows a recursive algorithmic loop. Agents first decompose the evaluation task, formulate preliminary assessments, and then enter a reflection phase where they consider the arguments of their peers. The Feedback Agent intervenes to resolve "inefficient dialogues," ensuring the agents do not merely agree to disagree but instead refine their logic toward a stable conclusion.

Datasets and Error Analysis

The researchers validated MATEval across four diverse datasets:

  • English: ROCStories (ROC) and WritingPrompts (WP).
  • Chinese: The Chinese LOng Text (LOT) dataset and a proprietary dataset from Alipay named "Ant," which focuses on fables and fairy tales.

To test diagnostic precision, the authors introduced five specific error types into a subset of 200 stories: Repetition (REP), Logical Inconsistency (LINC), Discontinuity (DCONT), Inappropriate Lexical Choice (ILC), and Factual Error (FER).

Key Results and Insights

Human Alignment

MATEval (SR+CoT) demonstrated superior performance compared to single-agent evaluators and traditional metrics. It achieved the highest Spearman ($\rho$) and Kendall ($\tau$) correlation with human scores across nearly all error categories. The framework proved particularly adept at identifying complex issues like Logical Inconsistency and Discontinuity, which are notoriously difficult for standard LLM evaluators to catch.

Diagnostic Depth and Industrial Utility

Unlike black-box scoring systems, MATEval generates a comprehensive diagnostic report. This report includes error localization, error classification, and detailed explanations. In a real-world deployment at Alipay, this transparency allowed developers to use "bad case" reports to immediately iterate on and fine-tune their models, significantly improving development efficiency.

Limitations and Future Directions

Despite its success, MATEval faces several challenges:

  • The Knowledge Gap: The framework's ability to detect Factual Errors is limited by the underlying LLM's internal knowledge. Future iterations may integrate Retrieval-Augmented Generation (RAG) to verify facts against external databases.
  • Language Nuances: Performance varied slightly between English and Chinese datasets, suggesting that multi-agent prompt engineering may need language-specific optimization.
  • Operational Costs: The multi-round nature of the discussion increases computational latency and API costs compared to single-pass evaluations, making it more suitable for offline model auditing than real-time applications.

Conclusion

MATEval represents a significant step forward in automated text evaluation. By shifting the focus from individual scoring to collaborative reasoning, the framework provides a robust, interpretable, and highly accurate method for assessing the creative outputs of modern LLMs.

Summary updated

Abstract

Recent advancements in generative Large Language Models (LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A “Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents’ interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.
BibTeX
@inproceedings{li2024mateval,
  author = {Yu Li and Shenyu Zhang and Rui Wu and Xiutian Huang and Yongrui Chen and Wenhao Xu and Guilin Qi and Dehai Min},
  title = {{MATEval: A Multi-agent Discussion Framework for Advancing Open-Ended Text Evaluation}},
  booktitle = {DASFAA},
  year = {2024},
  url = {https://openreview.net/forum?id=AoeIvOC3Qb},
}
Figure