Supervised Hierarchical Classification for Student Answer Scoring

Itziar Aldabe, Oier Lopez de Lacalle, Iñigo Lopez-Gazpio, Montse MaritxalararXiv 2015 · OpenReview

Abstract

This paper describes a hierarchical system that predicts one label at a time for automated student response analysis. For the task, we build a classification binary tree that delays more easily confused labels to later stages using hierarchical processes. In particular, the paper describes how the hierarchical classifier has been built and how the classification task has been broken down into binary subtasks. It finally discusses the motivations and fundamentals of such an approach.

Introduction

Automated Student Response Analysis (SRA) is critical for providing immediate feedback in Learning Management Systems. While binary grading (correct vs. incorrect) is relatively straightforward, real-world educational contexts require finer granularity. The Semeval-2013 SRA task introduced a five-way classification challenge: correct, partially_correct, irrelevant, non_domain, and contradictory.

Standard "flat" classifiers, which attempt to select one of five labels in a single step, struggle as categories increase. Macro-average F1 scores often drop from ~0.72 in binary tasks to ~0.42 in five-way tasks. This performance degradation occurs because labels are not mutually exclusive in a simple way; they exhibit "gradual relationships" where certain labels, like correct and partially_correct, are more easily confused than others.

Methodology

To address these challenges, we propose a Supervised Hierarchical Classification approach. The methodology consists of three primary stages:

  1. Measuring Label Relatedness: A standard five-way classifier is trained to generate a confusion matrix. Analyzing these errors allows for the calculation of a "distance" between labels based on frequency of confusion.
  2. Building the Hierarchy: Using agglomerative hierarchical clustering, we create a dendrogram that groups the most similar labels together, as shown in Figure 1.
  3. Binary SVM Tree: We implement a hierarchy of binary Support Vector Machines (SVMs). Instead of a single multi-class decision, the system executes a sequence of binary choices (e.g., "Is this answer non-domain or something else?").
Dendrogram showing the hierarchical clustering of labels based on similarity.

We evaluated two configurations: H1, which strips away the most dissimilar labels first (e.g., separating non-domain from the rest), and H2, which attempts to distinguish between the most confusing labels first (e.g., correct vs. partially_correct).

The study utilized the Semeval-2013 SRA dataset, comprising the BEETLE corpus (electronics tutorials) and the SCIENTSBANK corpus (science modules). Features included 30 syntactic and semantic similarity measures, such as text overlap and WordNet-based alignment.

Results and Discussion

Our analysis revealed that classification errors are highly structured, with 60% of all errors originating from just five specific label pairings. By imposing a tree structure, the hierarchical model acts as a regularizer, providing a "coherent bias" that prevents overfitting to multi-class noise.

Table of F-scores for the H1 and H2 hierarchical configurations at various levels of the tree.

As seen in Figure 2, the hierarchical approach achieves high precision at the upper nodes. In the H1 configuration, the first node (separating non-domain) achieved a near-perfect F-score of 0.988. However, the system faces the challenge of error propagation: a mistake at a high-level node cascades downward, preventing correct labeling by subsequent nodes. Despite this, the hierarchical H2 system achieved a Micro F1 of 0.426 on the final test set, outperforming the flat five-way SVM (0.415).

Conclusion

Hierarchical classification offers a promising path for nuanced student answer scoring by decomposing complex tasks into manageable binary decisions. Future work will focus on feature specialization, as the lexical features used to identify non-domain answers differ significantly from the deep semantic analysis required to distinguish correct from contradictory responses.

Summary updated

Abstract

This paper describes a hierarchical system that predicts one label at a time for automated student response analysis. For the task, we build a classification binary tree that delays more easily confused labels to later stages using hierarchical processes. In particular, the paper describes how the hierarchical classifier has been built and how the classification task has been broken down into binary subtasks. It finally discusses the motivations and fundamentals of such an approach.
BibTeX
@misc{aldabe2015supervised,
  author = {Itziar Aldabe and Oier Lopez de Lacalle and Iñigo Lopez-Gazpio and Montse Maritxalar},
  title = {{Supervised Hierarchical Classification for Student Answer Scoring}},
  year = {2015},
  eprint = {1507.03462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1507.03462},
}
Figure