Introduction
Automated essay grading has traditionally relied on "bag of words" models such as Latent Semantic Analysis (LSA). While effective at identifying topical relevance through word frequency and proximity, these models often fail to capture the complex causal relationships and subtle distinctions necessary for grading qualitative essays in scientific domains. In physics, for instance, the relationship between entities—such as whether a force causes motion or vice versa—is critical. Standard statistical models often miss these subtle but critical distinctions because they ignore syntax and functional relationships.
To address this, researchers developed CarmelTC, a hybrid text classification system. CarmelTC combines deep syntactic functional analysis (symbolic AI) with statistical machine learning (Naive Bayes) to better identify "correct answer aspects" in student writing.
Methodology
CarmelTC utilizes an ID3 decision tree learner to classify sentence segments based on a unique feature set derived from two distinct methodologies:
- Deep Syntactic Analysis (Symbolic): Using the CARMEL natural language processing tool, the system extracts features encoding functional relationships, such as subject-verb-object triples, verb tense, negation, and passivization. This allows the system to "factor out" surface-level variations, such as active versus passive voice, which do not alter the underlying physical meaning.
- Statistical Classification (Bag of Words): The system incorporates the output of a Rainbow Naive Bayes classifier as a feature, providing broad contextual information based on vocabulary.
The study evaluated these methods using the Pumpkin Problem Corpus, consisting of 126 student essays (500 segments) on a qualitative physics problem. The system was also trained on a development corpus of human-human tutoring dialogues and an LSA space derived from three introductory physics textbooks.
Results
The performance of CarmelTC was compared against LSA, a standalone Naive Bayes classifier (Rainbow), and a purely symbolic version of the tool (CarmelTCsymb).
The results demonstrate that while LSA achieved high precision (93%), its recall was significantly lower (54%), meaning it failed to identify nearly half of the correct points made by students. In contrast, the hybrid CarmelTC approach achieved the best balance, reaching a recall of 80% and a precision of 90%, resulting in the highest overall F-score of 0.85. Notably, the hybrid approach was significantly better at filtering out "false positives"—sentences that use physics words but do not actually express a key point.
Discussion and Conclusion
The success of CarmelTC highlights the importance of syntax in scientific domains. By extracting functional relationships, the system is more robust to surface variations in student writing and more effective at identifying the underlying logic of a student's argument.
While the deep analysis required by CarmelTC is more computationally intensive than simple word-counting methods, its superior accuracy makes it a promising tool for real-time dialogue systems, such as the Why2 tutorial system, which aim to provide structurally-aware feedback to students. Future work may involve expanding the system's grammar to cover a broader range of scientific domains beyond the specific physics problems tested.
Summary updated