TL;DR
This paper introduces KAES, a novel framework for cross-prompt automated essay scoring (AES) that goes beyond coarse-grained shared knowledge to discover and leverage detailed, multi-aspect shared knowledge. By using unsupervised clustering and LLMs to explicitly capture this knowledge, KAES significantly improves essay scoring accuracy on unseen prompts, demonstrating a more robust approach to automated essay evaluation.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) aims to automatically evaluate essays, reducing teacher workload and providing feedback. While prompt-specific AES focuses on essays from a single prompt, cross-prompt AES tackles the challenge of training a model on essays from one set of prompts and applying it to essays from entirely new prompts. This is crucial because obtaining human-rated essays for every new prompt is expensive and time-consuming. The core challenge in cross-prompt AES is learning "shared knowledge" that generalizes across different prompts. Existing methods often focus on coarse-grained, general shared knowledge, neglecting the finer, multi-faceted aspects of knowledge that truly underpin essay quality. This paper addresses this gap by proposing a method to automatically discover and utilize these detailed, multi-aspect shared knowledge signals.
Methods
KAES employs a two-pronged approach:
- Multi-aspect Knowledge Finding and Aligning:
- Knowledge Discovering: Essays from various source prompts are sampled and clustered using unsupervised methods (like K-Means or GMM). Each cluster is treated as a signal for a specific type of shared knowledge.
- Implicitly Knowledge Aligning: The model's encoder is trained to bring essay representations within the same cluster closer together. This is achieved by minimizing divergence between essay-to-cluster assignments, effectively aligning essays that share implicit knowledge.
- Explicitly Knowledge Capturing: To make the implicit knowledge interpretable, a Large Language Model (LLM, specifically ChatGPT) is used. Representative essays from each cluster are fed to the LLM to generate textual descriptions of the shared features. These descriptions are then encoded using BERT to create explicit knowledge representations.
- Knowledge Selection: For a given essay, the system dynamically selects the most relevant explicit knowledge representation based on the proximity of the essay's representation to the cluster centroids. This selected knowledge is then concatenated with the essay's original representation.
- Iterative Training Scheme: The model is trained in alternating iterations of multi-trait scoring and knowledge finding/aligning. This allows the encoder to be refined by the discovered shared knowledge, leading to improved essay representations for scoring.
Datasets
The experiments were conducted on the ASAP and ASAP++ datasets. These datasets contain English essays written in response to eight different prompts, with scores for multiple traits. The evaluation uses a prompt-wise validation method, common in cross-prompt AES research.
Key Insights and Ideas
- Beyond Coarse-Grained Knowledge: The paper convincingly argues that simply learning general shared knowledge isn't enough. Detailed, multi-aspect knowledge (e.g., specific grammatical nuances, coherence patterns) is crucial for robust cross-prompt generalization.
- LLMs for Interpretability: Leveraging LLMs to translate implicit, cluster-based knowledge into explicit, human-readable descriptions is a clever way to bridge the gap between raw data patterns and actionable insights. This also makes the learned knowledge more understandable.
- Dynamic Knowledge Selection: The ability to dynamically select the most relevant shared knowledge for each essay, rather than applying a one-size-fits-all approach, is key to KAES's effectiveness. This allows the model to adapt to the specific nuances of individual essays.
- Iterative Refinement: The alternating training of the scoring module and the knowledge discovery/alignment module creates a feedback loop that continuously improves the essay encoder's ability to capture and utilize shared knowledge.
Key Results
- KAES (specifically KAES(hier)+gmm) significantly outperforms state-of-the-art (SOTA) models like PLAES, achieving average improvements of 2.2% on prompts and 2.4% on traits.
- The method demonstrates substantial gains on specific prompts (up to 5.1% on P5) and traits (up to 5.7% on WC).
- Ablation studies show that removing the LLM-based knowledge capture (LLM) component leads to a decrease in performance by 0.9% on prompts and 1.1% on traits, highlighting the importance of explicit knowledge. Removing both the clustering (Cluster) and LLM components results in a larger performance drop.
- KAES shows good in-domain stability, meaning its performance on prompts it was trained on does not degrade significantly while improving cross-domain performance.
- The model achieves strong performance with a comparable number of parameters to other SOTA models, indicating good parameter efficiency.
Limitations and Future Work
- Computational Cost: The paper acknowledges that the clustering process introduces additional time costs. Training KAES(hier) takes 180 seconds per epoch, and KAES(bert) takes 1200 seconds per epoch. Memory usage can also be significant.
- Parameter Sensitivity: The paper briefly touches on the influence of the number of clusters and sampling quantity, noting that performance subtly varies with different cluster numbers (optimal at K=5) and that larger sample sizes for clustering lead to better results. Further exploration of these hyperparameter sensitivities could be beneficial.
- Future Work: The authors suggest that KAES is well-suited for other tasks like cross-domain sentiment classification and cross-lingual text classification, indicating potential for broader application.
Summary updated