ESAS: Towards Practical and Explainable Short Answer Scoring (Student Abstract)

Palak Goenka, Mehak Piplani, Ramit Sawhney, Puneet Mathur, Rajiv Ratn Shah

Abstract

Automated Short Answer Scoring (SAS) is a critical component for scaling modern education, yet existing systems often function as "black boxes," providing scores without pedagogical justification. This paper introduces ESAS (Explainable Short Answer Scoring), a novel framework that integrates lexical, syntactic, and semantic features with external context knowledge. By utilizing a tree-based ensemble architecture, ESAS provides both quantitative grades and qualitative feedback. Evaluated on the Automated Student Assessment Prize (ASAP) dataset, ESAS demonstrates a 7.8% improvement in Quadratic Weighted Kappa (QWK) over state-of-the-art models, particularly excelling in cross-prompt "combined model" scenarios.

Introduction

Short Answer Scoring (SAS) remains a significant bottleneck in educational technology. While multiple-choice assessments are easily automated, short-form written responses require a level of nuance that human graders struggle to provide consistently at scale. Human grading is frequently hindered by subjective bias, emotional state, and the sheer volume of submissions in environments like Massive Open Online Courses (MOOCs).

Existing automated systems often prioritize accuracy at the expense of transparency. These systems provide a score but fail to explain why a student received it or how they might improve. The ESAS framework aims to bridge this gap by creating a tool that is not only highly accurate but also inherently explainable, providing qualitative feedback alongside quantitative marks.

Methodology

The ESAS pipeline is designed as an end-to-end system consisting of three primary stages: Feature Extraction, Model Architecture, and Output Generation.

Feature Extraction

The system extracts five distinct categories of features to capture the complexity of student writing:

  • Lexical & Syntactic: Standard metrics including word count, sentence length, and Part-of-Speech (POS) tag distributions.
  • Readability & Cohesion: Metrics assessing the flow and linguistic complexity of the response.
  • Context Features: A novel approach using RAKE (Rapid Automatic Keyword Extraction) and PageRank on external sources like Wikipedia to build "context vectors." This allows the model to gain domain-specific knowledge without requiring retraining for every new subject.
  • Semantic Overlap: Utilizing BERT, Word2Vec, and Doc2Vec embeddings, the system measures the "concept overlap" between a student's response and a "gold standard" rubric.
The ESAS pipeline, illustrating the flow from feature extraction through the ensemble model to the dual-output scoring system.

Model Architecture

Rather than relying on opaque deep learning models, ESAS employs a Maximum Voting Ensemble of three tree-based statistical models: Random Forest, Gradient Boosting, and Extra-Trees. Tree-based models were selected for their robustness and high interpretability. By analyzing feature importance within these models, the system can pinpoint which specific linguistic elements (e.g., relevance, creativity, or readability) contributed most to a student's score.

Key Results

The system was evaluated using the ASAP dataset, a standard benchmark for SAS. Performance was measured using the Quadratic Weighted Kappa (QWK), which quantifies the agreement between automated and human graders.

Performance comparison (QWK) of ESAS against state-of-the-art models across ten ASAP datasets, highlighting the superior 'Overall' performance of the ESAS combined model.

As shown in the results, ESAS outperformed existing state-of-the-art models (Riordan et al., Ramachandran et al., and Kumar et al.) by 7.8% when utilizing a "combined model" approach—training on all prompts simultaneously rather than building prompt-specific models. This suggests that ESAS is more robust and truly domain-agnostic. Notably, the system achieved a 41% improvement on prompt-8, a complex comprehension task, compared to previous benchmarks. These gains were found to be statistically significant ($p < 0.05$) via Wilcoxon’s test.

Discussion and Conclusion

ESAS represents a shift toward practical, deployable automated grading. By integrating external knowledge and utilizing interpretable ensemble models, the framework provides a scalable solution that does not sacrifice pedagogical value for accuracy.

While the system currently relies on high-quality "gold standard" responses for its semantic overlap features, future work will focus on refining the system's performance in environments where such rubrics are unavailable. Additionally, the authors intend to conduct longitudinal studies to measure the actual effectiveness of the generated qualitative feedback on student learning outcomes. Ultimately, ESAS moves the field closer to an automated grading system that acts as a true partner in the educational process.

Summary updated

Abstract

Motivated by the mandate to design and deploy a practical, real-world educational tool for grading, we extensively explore linguistic patterns for Short Answer Scoring (SAS) as well as authorship feedback. We approach the SAS task via a multipronged approach that employs linguistic context features for capturing domain-specific knowledge while emphasizing on domain agnostic grading and detailed feedback via an ensemble of explainable statistical models. Our methodology quantitatively supersedes multiple automatic short answer scoring systems.
BibTeX
@inproceedings{goenka2020esas,
  author = {Palak Goenka and Mehak Piplani and Ramit Sawhney and Puneet Mathur and Rajiv Ratn Shah},
  title = {{ESAS: Towards Practical and Explainable Short Answer Scoring (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7170},
}
Figure