TL;DR
This paper introduces Roundtable Essay Scoring (RES), a novel multi-agent framework for automated essay scoring (AES) that leverages Large Language Models (LLMs) to achieve human-aligned scoring without task-specific training. By simulating a collaborative discussion among diverse LLM evaluators, RES significantly improves scoring accuracy and explainability, addressing a key challenge in educational assessment.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) has long been a goal in educational technology, aiming to provide efficient and reliable feedback on student writing. Traditional AES methods often struggle with capturing the nuanced, multi-faceted nature of human judgment, especially when it comes to higher-order thinking skills like reasoning and expression. While recent advances in LLMs have shown promise for zero-shot AES (evaluating essays without prior training), current approaches often rely on single LLMs or static prompting, failing to fully exploit the potential of LLMs for complex, multi-perspective evaluation. This limits their ability to achieve human-level alignment and provide truly insightful assessments. The challenge is to move beyond simply using LLMs as automated graders to leveraging their collaborative reasoning capabilities for more sophisticated evaluation.
Methods
The proposed Roundtable Essay Scoring (RES) framework operates in two main stages:
- Multi-Perspective Essay Evaluation:
- Evaluator Persona Creation: For a given essay and prompt, RES generates specialized LLM "evaluator agents." These agents are assigned distinct personas (e.g., English teacher, social studies teacher) tailored to the essay's topic and student proficiency level. This ensures a range of perspectives and prevents overly lenient or strict evaluations.
- Automated Rubric Construction: Each evaluator agent independently constructs a trait-based rubric relevant to its persona and the essay's context. These rubrics go beyond general writing traits to incorporate criteria specific to the essay's topic and purpose.
- Rationale-first Multi-trait Evaluation: Before assigning scores, each agent generates detailed rationales justifying its evaluation for each trait. This makes the agent's reasoning explicit and enhances explainability.
- Dialectical Reasoning:
- Dialectical Dialogue Simulation: The evaluator agents then engage in a simulated "roundtable discussion." They present their rationales, critique each other's assessments, and collaboratively refine their judgments. This process mimics human deliberation, allowing for consensus-building and deeper analysis.
- Moderator-led Dialectical Synthesis: A moderator agent synthesizes the discussion, resolves disagreements, and assigns a final holistic score. This ensures a well-justified and balanced outcome.
Datasets
The primary dataset used for evaluation is the ASAP dataset (Automated Student Assessment Prize). This dataset contains approximately 12,978 essays written by students in grades 7-10 in response to eight different prompts. The authors used 10% of the test split for evaluation.
Key Insights and Ideas
- Simulating Human Deliberation: The core innovation is using LLMs not just as individual scorers but as participants in a simulated dialectical reasoning process, mirroring how human experts might discuss and arrive at a consensus.
- Persona-Driven Evaluation: Tailoring LLM agents with specific personas and expertise leads to more nuanced and contextually relevant evaluations than a single, generic LLM.
- Rationale-First Approach: Requiring agents to provide rationales before scoring makes their evaluation criteria transparent and forms the basis for productive dialogue.
- Zero-Shot Advantage: The framework achieves strong performance without any task-specific fine-tuning, making it highly adaptable and practical for real-world educational settings where labeled data is scarce.
- Dialectical Reasoning Boosts Performance: The multi-agent dialogue and synthesis stage significantly improves scoring accuracy compared to individual agent evaluations or simpler prompting methods.
Key Results
Experiments on the ASAP dataset using ChatGPT and Claude models demonstrated significant improvements:
- RES achieved up to a 34.86% improvement in average Quadratic Weighted Kappa (QWK) over straightforward prompting (Vanilla) methods.
- With ChatGPT, RES achieved an average QWK of 0.483, outperforming Vanilla (0.364) and Multi-Trait Specialization (MTS) (0.412).
- With Claude, RES achieved an average QWK of 0.499, outperforming Vanilla (0.370) and MTS (0.304).
- An ablation study showed that increasing the number of agents improved performance, with diminishing returns after a certain point.
- The inclusion of Dialectical Reasoning (DR) was crucial, boosting performance by 20.6% on average compared to RES without DR.
Limitations and Future Work
- API Dependency and Cost: The current implementation relies on API-based proprietary LLMs (ChatGPT, Claude), which can be more costly than open-source alternatives. While experiments with open-source LLMs were conducted, they exhibited weaker instruction-following capabilities. Future work could explore adapting RES to open-source models more effectively.
- Focus on Scoring: The current framework focuses exclusively on scoring. LLMs are capable of generating feedback, and future work aims to expand RES to evaluate and enhance feedback quality, addressing aspects beyond grammar to content and structure.
- Bias in Commercial Models: The authors acknowledge that commercial LLMs may contain inherent biases due to their training data, and emphasize that the system should not be used for high-stakes educational decisions without human oversight.
- Real-time Performance: RES incurs greater computational cost and latency than single-prompt methods. While the performance gains justify this trade-off for many educational contexts, real-time applications might require further optimization.
Summary updated