TL;DR
This paper introduces a novel deep learning model for automated essay scoring (AES) that integrates Item Response Theory (IRT) to improve the interpretability of analytic score predictions. By combining the feature extraction power of deep neural networks with the psychometric rigor of multidimensional IRT, the model offers a more transparent understanding of how essays are scored across different dimensions, without significantly sacrificing accuracy.
What Problem Does This Paper Address?
Essay exams are valuable for assessing higher-order thinking skills, but manual grading is costly, time-consuming, and prone to subjective bias. Automated Essay Scoring (AES) offers a solution, but most existing deep learning AES models focus on predicting a single overall score. While analytic scores (e.g., for content, organization, word choice) are crucial for providing detailed feedback, current deep learning approaches for analytic scoring often result in complex, black-box neural architectures that are difficult to interpret. This lack of interpretability hinders understanding of why a particular score is assigned, making it challenging to trust or improve the system. This paper aims to bridge this gap by developing an AES model that not only predicts analytic scores accurately but also provides interpretable insights into the scoring process.
Methods
The core innovation of this paper is the integration of a Multidimensional Generalized Partial Credit Model (M-GPCM), a type of IRT model, into a deep neural network architecture for AES.
The proposed model builds upon a conventional deep learning AES architecture that includes:
- Shared Layer: Processes essay sentences through embedding, convolution, and attention pooling layers to create sentence-level representations. These are then aggregated into a fixed-length vector via a recurrent layer and another attention pooling layer.
- Item-Specific Layer: Takes the aggregated essay representation and, in the conventional model, uses complex, separate neural networks (trait attention, dense layers) for each analytic score.
The authors' novel contribution lies in replacing the complex item-specific layers with a multidimensional IRT layer. Specifically, they integrate the M-GPCM. This layer takes the essay-level vector from the shared layers and, using learned parameters, predicts the probability distribution of scores for each analytic item. The model is trained end-to-end using Categorical Cross-Entropy (CCE) loss.
The key advantages of this IRT integration are:
- Interpretability: The IRT parameters (discrimination, difficulty, and latent ability) are well-understood psychometric constructs, allowing for a more transparent explanation of score predictions.
- Ability Dimension Analysis: The multidimensional nature of the M-GPCM allows for the investigation and interpretation of latent ability dimensions underlying the analytic scores.
Datasets
The authors evaluated their model using two widely recognized datasets in AES research:
- ASAP (Automated Student Assessment Prize): This dataset, originating from a Kaggle competition, contains essays for eight prompts. Overall scores are available for prompts 1-6, with analytic scores also provided for prompts 7 and 8.
- ASAP++: This dataset supplements ASAP by providing analytic scores for prompts 1-6, which were previously only available as overall scores.
Key Insights and Ideas
- IRT as an Interpretability Layer: The paper demonstrates that IRT, a psychometric framework, can be effectively integrated into deep learning models to provide interpretable parameters (discrimination, difficulty, ability) for AES, moving beyond black-box predictions.
- Uncovering Latent Abilities: The multidimensional IRT component allows for the analysis of latent ability dimensions. Surprisingly, for the ASAP dataset, the analysis suggested that even with multiple analytic scores, often only one or two underlying ability dimensions were significantly measured.
- Word Input Superiority for Prompt-Specific Tasks: Contrary to some prior work that used Part-of-Speech (POS) tags for cross-prompt tasks, this study found that using raw word sequences as input yielded higher average performance for prompt-specific scoring.
- Expected Scores Outperform Argmax: Predicting the "expected score" (a weighted average of possible scores based on probabilities) generally led to better performance (higher QWK scores) than simply taking the most probable score (argmax).
Key Results
- Comparable Accuracy: The proposed IRT-integrated model achieved scoring accuracy (measured by Quadratic Weighted Kappa - QWK) comparable to the state-of-the-art conventional deep learning model, with only a slight degradation in performance. This is notable given the significantly fewer parameters in the IRT output layer compared to the complex item-specific layers of the conventional model.
- Interpretability Achieved: The authors successfully demonstrated how the IRT parameters (discrimination, difficulty) and latent ability estimates can be used to interpret score predictions and understand how different analytic scoring items measure examinee abilities. For instance, they showed how the "overall" score item might be less discriminative for a specific latent dimension compared to other analytic items.
- Optimal Dimension Analysis: PCA analysis on prompt 1 suggested a significant decrease in eigenvalues after the second component, indicating that a one- or two-dimensional latent ability model was sufficient to explain the data. While two- and three-dimensional models showed slightly better QWK scores, the differences were marginal.
Limitations and Future Work
- Dataset Specificity: The study was conducted on the ASAP and ASAP++ datasets. Future work should evaluate the model's performance on various other benchmark datasets to assess its generalizability.
- Cross-Prompt Tasks: The current model is primarily designed for prompt-specific scoring. An extension to handle cross-prompt tasks is a significant area for future research.
- Underlying Ability Dimensions: The finding that multiple analytic scores might only measure one or two latent abilities is an important insight but also raises questions about the design of analytic scoring rubrics. Future work could explore how to design rubrics that better capture distinct aspects of ability.
- Model Extension: The authors suggest developing extensions to their model for cross-prompt tasks, which would be a valuable next step.
Summary updated