Abstract
Automated grading in Intelligent Tutoring Systems (ITS) often relies on coarse-grained analysis, providing binary feedback that fails to identify specific knowledge gaps. This paper introduces a granular evaluation framework that decomposes reference answers into Minimal Meaningful Propositions (MMPs). By leveraging Deep Neural Networks (DNNs) and word embeddings to detect textual entailment between student responses and these MMPs, the proposed system significantly improves the identification of understood concepts compared to traditional methods. Experimental results on a real-world science classroom dataset demonstrate that a DNN approach with GloVe embeddings achieves a weighted average F1-score of 0.76, outperforming complex feature-engineered models.
Introduction
Most current Intelligent Tutoring Systems (ITS) perform "coarse-grained" analysis, comparing a student’s response to a teacher’s reference answer as a whole. However, a reference answer often contains multiple distinct concepts. If a student understands two out of three points, a coarse-grained system might simply mark the answer as "partially correct" without explaining what was missing. This lack of specificity hinders the pedagogical value of automated feedback.
The challenge in automating this process lies in the nature of student data: responses are often ungrammatical, informal, and riddled with spelling errors. Furthermore, "Recognizing Textual Entailment" (RTE)—determining if one sentence logically follows from another—is a notoriously difficult Natural Language Processing (NLP) task. This research aims to bridge the gap between high-level grading and specific, actionable feedback by breaking down complex answers into their constituent claims.
Methodology
The researchers developed a two-step pipeline to achieve granular feedback:
1. Decomposition into Minimal Meaningful Propositions (MMPs)
The first step involves breaking the teacher's reference answer into Minimal Meaningful Propositions (MMPs). These are the smallest possible claims that retain contextual meaning. By evaluating student responses against these individual units, the system can pinpoint exactly which parts of a concept a student has mastered.
2. Entailment Detection
The system then checks if a student's response "entails" (matches the meaning of) each MMP. Two primary machine learning architectures were tested:
- SVM with Hand-Crafted Features: A support vector machine using 183 features, including BLEU scores, Pointwise Mutual Information (PMI), and semantic "facets" (governors and modifiers).
- Deep Neural Network (DNN): A feed-forward network using pre-trained GloVe word embeddings. The MMP and student response were represented by averaging their word vectors; the model then calculated the element-wise product and absolute difference between them to capture similarity.
To handle "noisy" student text, a custom preprocessing phase included an automatic spell checker that prioritized words found in the question or other students' responses.
Dataset
The study utilized a dataset from real-world middle school science classrooms, comprising 266 questions and 992 reference MMPs. This resulted in 20,815 entailment pairs. Pairs were labeled as Understood (31%) or Not Understood (69%).
Results and Discussion
The experimental results demonstrated that granularity is key for effective feedback. By treating an answer as a collection of MMPs, the system can recognize specific knowledge gaps—for instance, identifying that a student knows "goggles protect eyes" but forgot that "gloves protect hands."
Key insights from the evaluation include:
- Embeddings Outperform Manual Engineering: Despite a relatively small dataset for deep learning (20k instances), the DNN with 50-dimensional GloVe embeddings outperformed the highly complex SVM with 183 hand-tuned features.
- Improved "Understood" Detection: Identifying what a student does know is significantly harder than identifying what they don't. The DNN approach showed a 61% improvement in identifying the Understood class compared to previous state-of-the-art methods.
- Robustness to Noise: The contextual spell-checking approach proved vital for handling domain-specific science terms that standard spell checkers often fail to recognize.
Conclusion
This research provides a path toward automated feedback that tells a student exactly which parts of a concept they missed, rather than just providing a binary grade. While the current system relies on a single reference answer and faces challenges with class imbalance (specifically the rare "Misunderstood" category), it represents a significant step forward in educational NLP. Future work will explore using multiple valid reference answers and classifying question types to apply more specialized entailment logic.
Summary updated