TL;DR
This paper introduces GAT-AES, a novel graph-attention network approach to multi-trait essay scoring. By modeling essays as graphs where traits are nodes, GAT-AES effectively captures complex interdependencies between traits, leading to state-of-the-art performance on the ASAP++ dataset. This work shifts the paradigm from sequence-based essay analysis to a more relational graph-based understanding.
What Problem Does This Paper Address?
Traditional Automated Essay Scoring (AES) models typically treat essays as linear sequences of words. While this has been effective for holistic scoring (assigning a single overall score), it falls short for multi-trait scoring, where essays are evaluated on multiple dimensions (e.g., organization, content, style). Existing multi-trait models struggle to effectively capture the intricate relationships between these traits. Some models impose an artificial order on trait prediction, limiting information flow, while others use attention mechanisms that only capture one-hop dependencies. This limits the ability to understand how different aspects of an essay's quality influence each other, hindering both the accuracy of scoring and the richness of feedback provided to users.
Methods
The core innovation of this paper is the Graph-Based Multi-Trait Essay Scoring (GAT-AES) framework. Instead of a word sequence, an essay is represented as a graph.
- Graph Construction:
- Nodes: The graph includes three types of nodes:
- Text Embedding Nodes: Multiple nodes representing semantic information from different parts of the essay text, derived from a pre-trained language model (like BERT).
- Hand-crafted Feature Nodes: Nodes representing statistical and linguistic features extracted from the essay.
- Essay Trait Nodes: Nodes representing each trait to be scored (including the overall score).
- Edges: Three types of edges connect these nodes:
- Embedding-Trait Edges: Connect text embedding nodes to trait nodes, allowing traits to interact with essay content.
- Feature-Trait Edges: Connect hand-crafted feature nodes to trait nodes, enabling traits to leverage explicit linguistic features.
- Trait-Trait Edges: Connect every trait node to every other trait node, forming a fully connected subgraph of traits. This is the key to capturing inter-trait dependencies.
- Graph Attention Network (GAT): The constructed graph is then processed by a GAT. GAT layers use a multi-head self-attention mechanism to aggregate information from neighboring nodes, allowing each trait node to learn from other traits, essay content embeddings, and hand-crafted features. This enables the model to capture multi-hop dependencies between traits.
- Trait Scoring: After passing through several GAT layers, each trait node's final representation is fed into a regression head to predict the trait score. The model is trained end-to-end using Mean Squared Error (MSE) loss.
Datasets
The primary dataset used is ASAP++, an extension of the ASAP corpus. ASAP++ is notable for providing essays annotated with both holistic scores and scores for multiple traits. The authors also mention the ICLE++ corpus in passing, where similar trait correlations were observed.
Key Insights and Ideas
- Essay as a Graph: The fundamental shift from a linear sequence to a graph representation is a novel perspective for essay scoring, enabling a more holistic understanding of trait interactions.
- Trait Interdependencies are Crucial: The paper strongly demonstrates that traits are not independent. Positive correlations between trait pairs (PCC of at least 0.6) highlight the need for models that can explicitly learn these relationships.
- Multi-Hop Dependencies Matter: GAT's ability to capture multi-hop dependencies between traits is shown to be superior to methods that only consider one-hop interactions.
- Graph Structure is Key: Ablation studies reveal that removing trait-trait edges or reducing the number of embedding/feature nodes significantly degrades performance, underscoring the importance of the proposed graph structure.
- BERT's Effectiveness: While newer LMs were tested, BERT Large still provided competitive or superior performance when used with the GAT framework, suggesting that the graph architecture is a significant contributor to performance gains.
Key Results
- State-of-the-Art Performance: GAT-AES achieved the best multi-trait scoring results on the ASAP++ dataset, outperforming existing methods, including other graph-based approaches and models using trait-attention mechanisms.
- Improved QWK Scores: The paper reports significant improvements in Quadratic Weighted Kappa (QWK) scores across various traits, with GAT-AES achieving state-of-the-art for OVERALL, CONTENT, ORGANIZATION, WORD CHOICE, VOICE, and the average trait score.
- Consistency and Robustness: GAT-AES demonstrated lower standard deviations across cross-validation folds compared to other models, indicating more consistent and robust performance.
- Comparison with GNNs: GAT-AES outperformed other GNN baselines (GCN, GraphSAGE, GIN) on the essay scoring task, suggesting that GAT's specific attention mechanism is well-suited for capturing trait interdependencies in this domain.
Limitations and Future Work
- Computational Constraints: The authors acknowledge that they were limited to using language models with fewer than 5B parameters due to computational constraints and did not explore larger models.
- Rubric Information: The models did not incorporate information directly from the scoring rubrics, which the authors suggest could be beneficial for future work, especially for capturing subtle distinctions between score levels.
- Future Directions:
- Exploring larger language models.
- Incorporating rubric information into scoring models (e.g., through rubric-aware features or encoding score definitions).
- Applying graph-based representations to other AES tasks, such as cross-prompt scoring.
Summary updated