Real-Time Automated Answer Scoring

Akash Nagaraj, Mukund Sood, Gowri SrinivasaIcalt 2018 · DOI

TL;DR

This paper introduces a real-time automated answer scoring system that goes beyond evaluating final submissions by capturing "snapshots" of student progress. By leveraging deep learning with Word2Vec embeddings and an LSTM model, they achieve state-of-the-art performance, offering valuable insights into student thought processes and easing teacher workload.

What Problem Does This Paper Address?

The traditional method of grading constructed-response assessments, like essays, is time-consuming and labor-intensive for educators. This bottleneck limits the frequency with which teachers can assign writing tasks, hindering student development. While automated essay scoring (AES) systems exist, they often focus solely on the final answer. This paper addresses the need for an AES system that not only scores answers efficiently but also provides a real-time view of a student's writing process, offering deeper insights into their learning and potential misconceptions.

Methods

The core of this work lies in a deep learning approach that utilizes Word2Vec for word embeddings and a Long Short-Term Memory (LSTM) network for sequential data processing. Unlike previous methods that relied on handcrafted features, this approach learns representations directly from the text.

  • Word Embeddings: The authors experimented with both Word2Vec and GloVe. They found Word2Vec to be superior, especially when trained on their specific dataset, as it captures context-specific word meanings better than pre-trained GloVe vectors.
  • Deep Learning Models: They evaluated several models, including Random Forests, Support Vector Machines (SVM), Deep Neural Networks (DNN), and LSTMs. The LSTM, being a recurrent neural network with a memory unit, proved most effective for sequential text data, capturing long-term dependencies.
  • Real-Time "Snapshots": A key innovation is the system's ability to capture intermediate states of a student's answer as they type. This allows for the visualization of score progression over time, offering a window into the student's thought process.
  • Combined Model: To further enhance performance, they developed a combined model that takes a weighted average of predictions from individual models, leading to improved accuracy.
  • Pre-processing: They explored stemming and spell-checking but found they negatively impacted results, attributing this to the importance human graders place on grammar and word choice.

Datasets

The primary dataset used is The Hewlett Foundation: Automated Essay Scoring [4]. This dataset consists of:

  • 8 different essay sets, each from a single prompt.
  • Essays ranging from 150 to 550 words.
  • Human-graded scores for each essay.
  • A total of 10,686 samples, split into 85:15 for training and testing/validation.
  • Each set has a different grading scale.

Key Insights and Ideas

  • Contextual Word Embeddings Matter: Word2Vec, trained on the specific domain of student essays, significantly outperformed pre-trained GloVe embeddings. This highlights the importance of domain-specific representations for nuanced understanding.
  • Sequential Models Excel at Text: LSTMs, with their ability to process sequential data and maintain context, are inherently well-suited for understanding and scoring written text compared to static models like DNNs or traditional classifiers.
  • Process Over Product: Capturing intermediate "snapshots" of student writing offers a richer understanding of their learning journey than just evaluating the final output. This can reveal how students arrive at their answers, including potential backtracking or revisions.
  • Ensemble Learning for Robustness: Combining predictions from multiple models (LSTM, DNN, etc.) through weighted averaging leads to a more robust and accurate scoring system.

Key Results

  • The Combined Model achieved the highest Quadratic Weighted Kappa (QWK) score of 0.9721 with 300-dimensional Word2Vec embeddings.
  • The LSTM model alone also showed strong performance, outperforming the Deep Neural Network.
  • The system can predict scores for "snapshots" of a student's answer in approximately 0.3 seconds, enabling a real-time interactive experience.
  • The paper claims their model exceeded previous prediction models by a large margin compared to competitors in the Kaggle competition.

Limitations and Future Work

  • Pre-processing Impact: While stemming and spell-checking were explored, they were found to reduce accuracy, suggesting that the nuances of language and grammar are critical for human-like scoring.
  • Real-world Data: The use of human-graded data is a strength, but the paper acknowledges that automated scoring is intended as an aid to human evaluators, not a complete replacement.
  • Future Work: The authors suggest that the insights gained from real-time progress tracking could lead to a better understanding of student thought processes, potentially informing teaching strategies and curriculum development. Further exploration into more sophisticated pre-processing techniques or model architectures could also yield improvements.

Summary updated

BibTeX
@inproceedings{nagaraj2018real,
  author = {Akash Nagaraj and Mukund Sood and Gowri Srinivasa},
  title = {{Real-Time Automated Answer Scoring}},
  booktitle = {Icalt},
  year = {2018},
  doi = {10.1109/icalt.2018.00122},
}
Figure