TL;DR
This paper tackles the challenge of automated essay scoring (AES) where models trained on one set of essay prompts need to perform well on unseen prompts. They introduce a novel "scoring-invariance" objective to encourage models to learn essay features that are inherently indicative of quality, regardless of the prompt. This, combined with a routing gate mechanism for better trait-specific feature extraction and a trait consistency objective, significantly improves cross-prompt scoring performance, especially in low-resource scenarios.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) systems are valuable for providing feedback on various essay aspects like quality, organization, and prompt adherence. Traditionally, AES models are trained and tested on essays from the same prompt. However, in real-world scenarios, obtaining labeled essays for every new prompt is impractical. This leads to the problem of cross-prompt essay scoring, where models must generalize to score essays from prompts they haven't seen during training. Existing approaches often focus on learning "prompt-invariant" representations, aiming to make models robust to prompt variations. This paper argues that this approach might be too broad and proposes a more focused "scoring-invariance" perspective, emphasizing features that intrinsically reflect essay quality, rather than just being invariant to prompt changes. They also address the challenge of multi-trait scoring, where essays are evaluated across several dimensions simultaneously, requiring models to capture diverse yet consistent trait-specific information.
Methods
The core of the proposed method, named Scoring-invariance Enhanced Cross-prompt Trait Scoring (SICTS), lies in its novel optimization objectives and architectural enhancements:
- Scoring-Invariance Learning Objective: Instead of just aiming for prompt-invariance, this objective encourages the model to focus on features that are consistently indicative of essay quality, regardless of whether they are derived from prompt-aware or prompt-agnostic representations. This is achieved by minimizing the distance between these two types of representations.
- Sentence-Level Routing Gating Network: To better capture diverse trait-specific features in multi-trait scoring, a "mixture-of-experts" (MoE)-inspired routing gate mechanism is introduced at the sentence level. This allows different trait-scoring modules to selectively focus on essay features most relevant to their specific trait, enhancing feature diversity.
- Trait Consistency Scoring Objective: To further improve multi-trait scoring, a trait consistency loss is proposed. This objective encourages alignment of essay traits in the latent space and also considers the consistency between individual traits and the overall essay score.
- Hierarchical Architecture: The model uses a standard hierarchical structure (CNN + Attention for sentence encoding, LSTM + Attention for document encoding) to process essays.
Datasets
The experiments were conducted on the ASAP (Automated Student Assessment Prize) and ASAP++ datasets. These datasets are standard benchmarks for AES and contain essays from multiple prompts, each with various scoring dimensions. The paper does not introduce new datasets.
Key Insights and Ideas
- Scoring-Invariance is More Focused: The paper posits that "scoring-invariance" (features inherently reflecting quality) is a more effective target for cross-prompt AES than general "prompt-invariance." This suggests that focusing on the essence of quality is key to generalization.
- Trait Diversity via Gating: The sentence-level routing gate mechanism effectively allows different parts of the model to specialize in different essay traits, leading to richer, more diverse trait representations.
- Low-Resource Performance Boost: The proposed method shows significant improvements in low-resource scenarios, where training data for the target prompt is scarce, highlighting its robustness.
- Generalization Enhances Trait Scoring: The study suggests that improving the model's general essay representation ability is crucial for better cross-prompt trait scoring.
Key Results
- State-of-the-Art Performance: SICTS achieved the best average Quadratic Weighted Kappa (QWK) scores across most prompts and traits compared to several strong baselines, including ProTACT and PLAES.
- Significant Improvement on Low-Resource Prompts: The model demonstrated a substantial improvement (e.g., 5.2% on prompt 8) in scoring performance on prompts with limited training data.
- Efficiency Gains: SICTS achieved better scoring performance with a significantly reduced model parameter count (nearly 33% fewer) compared to the SOTA method ProTACT.
- Ablation Studies Confirm Effectiveness: Ablation studies showed that the combination of scoring-invariance learning (SIL) and trait consistency loss (TCL) yielded the best results, with SIL proving particularly beneficial for generalization.
- LLMs Not Always Superior: A comparison with GPT-3.5-turbo in a zero-shot setting indicated that domain-specific small models like SICTS can still hold considerable value and even outperform large language models in this specific task.
Limitations and Future Work
- Dataset Specificity: The model's effectiveness was demonstrated on ASAP and ASAP++ datasets. Further testing on other datasets and languages is needed to confirm broader applicability.
- Further Generalizability Exploration: The authors suggest that continued exploration of methods to enhance the generalizability of essay representations could further improve cross-prompt scoring capabilities.
- Hyperparameter Sensitivity: While the paper explores hyperparameter settings, further investigation into their impact on generalization across diverse scenarios could be beneficial.
Summary updated