Get Semantic With Me! The Usefulness of Different Feature Types for Short-Answer Grading

Ulrike PadóCOLING 2016

TL;DR

This paper investigates which linguistic features are most effective for automatically grading short answers, finding that simpler lexical and similarity features often suffice, while deeper semantic features are crucial for more complex, paraphrased answers. The research provides a valuable guide for building efficient and effective automated grading systems by highlighting feature hierarchies and corpus-specific needs.

What Problem Does This Paper Address?

Automated grading of short-answer questions is a critical bottleneck in scaling educational assessments, especially with the rise of online learning. While NLP offers potential solutions, the field has proposed a wide array of linguistic features, leading to a lack of consensus on which ones are truly effective and generalizable. This paper aims to demystify this by investigating the relative importance of different feature types across various corpora and languages, seeking to establish a hierarchy of feature usefulness and understand how corpus characteristics influence optimal feature selection.

Methods

The authors employ a decision tree (J48 implementation) as their primary machine learning model, chosen for its ability to handle overfitting through pruning. They evaluate a comprehensive set of features derived from five levels of linguistic processing:

  • NGram features: Measure overlap of unigrams, bigrams, and trigrams (both tokens and lemmas) between student and reference answers.
  • Similarity features: Employ algorithms like Greedy String Tiling (GST), Cosine similarity, and Levenshtein edit distance to compare texts at a surface level. They explore variations with and without stop word removal, and with question word demotion.
  • Dependency features: Capture overlap in lemmatized dependency triples (governor, dependency type, dependent).
  • Semantics features: Utilize abstract semantic representations (LRS formalism) and alignment components to analyze meaning.
  • Textual Entailment (TE) features: Leverage TE systems to determine if the student answer is entailed by the reference answer, using both the decision and confidence score.

The core methodology involves computing these features in relation to a reference answer and then using them to train a classifier to predict whether a student's answer is correct or incorrect. They use leave-one-question-out cross-validation for smaller corpora and held-out test sets for larger ones.

Datasets

The study utilizes five corpora, chosen to represent different languages and elicitation tasks:

  • SciEntsBank (SEB) and Beetle: SemEval-2013 corpora, containing English content assessment questions from science instruction.
  • CSSAG: A set of German content assessment questions about programming in Java.
  • CREG: A German language-skill corpus where learners answer questions about a text.
  • CREE: An English language-skill corpus with similar characteristics to CREG.

These datasets are described as containing question texts, at least one reference answer, and student answers with human-assigned correctness judgments. For CSSAG, which uses a point-based scoring system, answers were converted to binary labels (correct/incorrect) based on a 50% threshold.

Key Insights and Ideas

  • Lexical and Similarity Features are Workhorses: Simpler features like NGrams and text similarity measures are consistently strong predictors of answer accuracy across most corpora. This explains why breaking the "lexical baseline" is so challenging in this field.
  • Deeper Features Shine for Paraphrasing: Features from deeper linguistic processing (semantics and TE) become significantly more useful when student answers exhibit substantial paraphrasing or semantic variation, particularly in content assessment corpora.
  • Language Doesn't Matter (Much): The study found little discernible difference in feature performance between German and English corpora, suggesting that language-specific adaptations (like lemmatization for German) are helpful but don't fundamentally alter feature hierarchies.
  • Corpus Type is Key: A clear difference emerged between content assessment corpora and language-skill corpora. Language-skill corpora benefit less from deeper features, as answers are often lifted directly from source texts and are lexically similar.
  • Question Word Demotion Can Backfire: While intended to focus on novel information, question word demotion can be counterproductive if the question itself is ambiguous or if correct answers happen to share many words with the question.
  • Feature Inter-correlation is High: Many features are highly correlated, meaning that combining them doesn't always yield proportional performance gains. Simpler features often capture much of the information that more complex features provide.

Key Results

  • The proposed feature set and decision tree model achieve realistic performance, approximating state-of-the-art results on most datasets.
  • For content assessment corpora (Beetle, CSSAG), features from deeper processing levels (semantics, TE) are highly predictive.
  • For language-skill corpora (CREG, CREE), NGram and similarity features are very strong predictors, and deeper features offer less additional benefit.
  • In four out of five corpora, subsets of feature groups were sufficient to approximate full model performance, and in some cases, even optimize it by using fewer features.
  • The study found that for some corpora, combining NGram and similarity features alone could reach or exceed full model performance.

Limitations and Future Work

  • SEB Data Anomaly: The SEB dataset showed an anomaly where the model underperformed against baselines, which the authors hypothesize is due to a mismatch between training and test data. They still include it for analysis due to consistent cross-validation results.
  • Feature Inter-correlation: The high inter-correlation between features suggests that while many features are useful, their additive value might be limited. Future work could explore more sophisticated feature selection or ensemble methods that account for this.
  • Extrinsically-Motivated Model Building: The authors suggest that future work could explore adding features in order of processing effort to build models incrementally, aiming for efficiency.
  • Corpus-Specific Recommendations: The findings provide a basis for building well-motivated models tailored to specific corpus types, suggesting further research could refine these recommendations.

Summary updated

BibTeX
@inproceedings{pado2016get,
  author = {Ulrike Padó},
  title = {{Get Semantic With Me! The Usefulness of Different Feature Types for Short-Answer Grading}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure