Automatic and Human Scoring of Word Definition Responses

Kevyn Collins-Thompson, Jamie CallanNAACL 2007 · OpenReview

Kevyn Collins-Thompson and Jamie Callan
Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics

Abstract

This paper introduces an automated method for scoring "definition production tests"—short, free-text responses where language learners attempt to define a target word. By using a Markov chain model on a graph of semantic word relations, the authors' system successfully bridges the vocabulary gap between idiosyncratic student answers and formal reference definitions, significantly outperforming standard text-similarity baselines.

Introduction

In language learning, assessing a student's vocabulary acquisition often involves "definition production tests." In these tests, a student writes a short phrase or sentence defining a newly learned word. Traditionally, these free-text responses must be manually graded by experts who compare them to a reference definition.

Manual scoring is time-consuming, expensive, and inflexible, making it a major bottleneck for real-time Computer Assisted Language Learning (CALL) systems. Furthermore, standard automated text-matching algorithms struggle with this task because student responses are very short (1–10 words) and often use entirely different vocabulary than the formal dictionary definition. For example, if the target word is ameliorate (reference: "to improve something"), a student might write "make it better." A system looking only for exact word overlap will score this as a zero, missing the fact that the student actually understands the concept.

Examples of student responses and their corresponding human-assigned scores.

Methods: Probabilistic Semantic Smoothing

To solve the vocabulary mismatch problem, the authors propose a probabilistic model of text semantic similarity based on a random walk (Markov chain) over a graph of word relations. Instead of relying purely on exact word overlap, the authors build a graph where nodes are words and edges represent various semantic relationships. They combine both corpus-based and knowledge-based resources to create these links:

  • Stemming: Morphological roots via the Porter stemmer.
  • Synonyms & Hypernyms: Structured dictionary relationships via WordNet.
  • Co-occurrence: Statistical likelihood of words appearing together in a corpus.
  • Free Association: Empirical human psychology data on word associations.

To score a student's response, the algorithm "lights up" the nodes corresponding to the words the student wrote and simulates a short random walk (up to 5 steps) across the graph's edges. The final score is based on the probability that this random walk lands on the specific words comprising the teacher's reference definition. This effectively performs "semantic smoothing," rewarding responses that use related concepts even if they share no exact words with the target.

Evaluation and Key Results

The system was evaluated using a dataset of 734 free-text responses to definition production tests, collected from a word learning experiment at the University of Pittsburgh. The dataset was manually scored by human coders on a 0–3 scale, with inter-rater reliability (weighted kappa) ranging from 0.64 to 0.72.

Ranking Quality

The Markov chain method achieved a Spearman Rank Correlation of 0.6111 with human judgments, significantly outperforming the Cosine baseline (0.4731) and Latent Semantic Analysis (LSA) (0.4868). A linear combination of LSA + Markov slightly improved this to 0.6365, although a gap remains compared to the human upper bound of 0.8744.

Spearman Rank Correlation results comparing various automated methods against human judgments.

Label Accuracy

When identifying the most accurate responses (measured by Root Mean Squared Error of the top item), the Markov model achieved the lowest error (0.7222) among automated methods, compared to Cosine (0.8194) and LSA (0.8009). This capability for granular assessment is highly valuable for tracking incremental learning in educational settings, where a student might grasp the general meaning of a word before mastering its exact definition.

Conclusion and Future Work

This research demonstrates that semantic smoothing via graph-based random walks is crucial for assessing short, idiosyncratic student texts. By integrating structured knowledge with empirical human data and statistical corpora, the model captures a richer representation of meaning than purely unsupervised methods.

However, limitations remain. The current model struggles with multi-word expressions and idioms (e.g., "to get away with" vs. "to get away from"), which simple word-graph models can easily conflate. Future work will focus on fine-tuning parameter estimation for link-specific decay rates and integrating these automated metrics into intelligent language tutors to enable real-time, adaptive lesson planning based on the nuance of student answers.

Summary updated

Abstract

Kevyn Collins-Thompson, Jamie Callan. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference. 2007.
BibTeX
@inproceedings{collinsthompson2007automatic,
  author = {Kevyn Collins-Thompson and Jamie Callan},
  title = {{Automatic and Human Scoring of Word Definition Responses}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
  url = {https://openreview.net/forum?id=B1-BLmZu-r},
}
Figure