A Hybrid Approach to Content Analysis for Automatic Essay Grading

Carolyn P. Rosé, Antonio Roque, Dumisizwe Bhembe, Kurt VanLehn

Introduction

Automated essay grading has traditionally relied on "bag of words" models such as Latent Semantic Analysis (LSA). While effective at identifying topical relevance through word frequency and proximity, these models often fail to capture the complex causal relationships and subtle distinctions necessary for grading qualitative essays in scientific domains. In physics, for instance, the relationship between entities—such as whether a force causes motion or vice versa—is critical. Standard statistical models often miss these subtle but critical distinctions because they ignore syntax and functional relationships.

To address this, researchers developed CarmelTC, a hybrid text classification system. CarmelTC combines deep syntactic functional analysis (symbolic AI) with statistical machine learning (Naive Bayes) to better identify "correct answer aspects" in student writing.

Methodology

CarmelTC utilizes an ID3 decision tree learner to classify sentence segments based on a unique feature set derived from two distinct methodologies:

  1. Deep Syntactic Analysis (Symbolic): Using the CARMEL natural language processing tool, the system extracts features encoding functional relationships, such as subject-verb-object triples, verb tense, negation, and passivization. This allows the system to "factor out" surface-level variations, such as active versus passive voice, which do not alter the underlying physical meaning.
  2. Statistical Classification (Bag of Words): The system incorporates the output of a Rainbow Naive Bayes classifier as a feature, providing broad contextual information based on vocabulary.

The study evaluated these methods using the Pumpkin Problem Corpus, consisting of 126 student essays (500 segments) on a qualitative physics problem. The system was also trained on a development corpus of human-human tutoring dialogues and an LSA space derived from three introductory physics textbooks.

Results

The performance of CarmelTC was compared against LSA, a standalone Naive Bayes classifier (Rainbow), and a purely symbolic version of the tool (CarmelTCsymb).

Performance comparison of different text classification approaches.

The results demonstrate that while LSA achieved high precision (93%), its recall was significantly lower (54%), meaning it failed to identify nearly half of the correct points made by students. In contrast, the hybrid CarmelTC approach achieved the best balance, reaching a recall of 80% and a precision of 90%, resulting in the highest overall F-score of 0.85. Notably, the hybrid approach was significantly better at filtering out "false positives"—sentences that use physics words but do not actually express a key point.

Discussion and Conclusion

The success of CarmelTC highlights the importance of syntax in scientific domains. By extracting functional relationships, the system is more robust to surface variations in student writing and more effective at identifying the underlying logic of a student's argument.

While the deep analysis required by CarmelTC is more computationally intensive than simple word-counting methods, its superior accuracy makes it a promising tool for real-time dialogue systems, such as the Why2 tutorial system, which aim to provide structurally-aware feedback to students. Future work may involve expanding the system's grammar to cover a broader range of scientific domains beyond the specific physics problems tested.

Summary updated

Abstract

We present CarmelTC, a novel hybrid text classification approach for automatic essay grading. Our evaluation demonstrates that the hybrid CarmelTC approach outperforms two "bag of words" approaches, namely LSA and a Naive Bayes, as well as a purely symbolic approach.
BibTeX
@inproceedings{rose2003hybrid,
  author = {Carolyn P. Rosé and Antonio Roque and Dumisizwe Bhembe and Kurt VanLehn},
  title = {{A Hybrid Approach to Content Analysis for Automatic Essay Grading}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure