Erantzunen kalifikazio automatikorako lehen urratsak

Eneko Agirre, Itziar Aldabe, Oier López de Lacalle, Iñigo López-Gazpio, Montse MaritxalarEKAIA 2015 · DOI

Abstract: Lan honetan erantzunen kalifikazio automatikorako lehen urratsak azaltzen ditugu. Urrats horiek azaltzeko irakaskuntzaren domeinuko bi sistema deskribatzen ditugu: bata, erantzunak hierarkikoki antolatzen eta kalifikatzen dituena; eta, bestea, kalifikazioaren azalpena emateko asmoz, erantzunaren zatiak erreferentzia baten aurka alderatzeko gai dena. Horretaz gain, irakaskuntzaren alorreko ataza internazional bat aztertuz alorreko teknika erabilienak mahairatzen ditugu.

Introduction

In the field of Educational Technology (EdTech), the development of systems capable of grading open-ended student answers with human-like accuracy is a significant challenge. While multiple-choice assessments are easily automated, they often fail to capture the nuance of a student's conceptual understanding. This research explores Automated Short Answer Grading (ASAG) within a Basque linguistic context, addressing two primary hurdles: the classification of nuanced student errors and the need for explainable feedback.

Methodology

The researchers propose two complementary approaches to move beyond "black box" grading systems:

1. Hierarchical EHU-ALM

Most existing systems treat grading as a flat classification task. However, human logic often groups similar types of errors. The EHU-ALM system was adapted into a hierarchical model using clustering based on a confusion matrix. This approach transforms the problem into a decision tree, allowing the system to first make broad distinctions—such as whether an answer is within the correct domain—before attempting to distinguish between finer categories like "Correct" and "Partially Correct."

Hierarchical decision tree for student response classification.

2. Phrase-Level Alignment

To improve explainability, a second system was developed to decompose sentences into chunks (phrases). Utilizing the Hungarian Algorithm, the system finds the optimal mapping between student phrases and reference answer phrases. This granular alignment allows the system to identify exactly which components of a student's response are missing or incorrect, providing a foundation for intelligent tutoring.

Evaluation and Results

The systems were evaluated using benchmarks from the SemEval (Semantic Evaluation) workshops, specifically the 2013 Joint Student Response Analysis and the 2015 Semantic Textual Similarity tasks.

In the classification task, the hierarchical version of the EHU-ALM system demonstrated a performance improvement over the standard flat classifier. As shown in the results table below, the hierarchical approach achieved an F-score of 0.426 compared to the baseline of 0.415.

Comparison of F-scores between the baseline (Oinarrizko sistema) and hierarchical (Sistema hierarkikoa) systems.

Furthermore, the segment-alignment system performed exceptionally well in the SemEval 2015 task, achieving a score of 0.8846, which significantly outperformed the task average of 0.8193 and nearly matched the top-performing international system.

Conclusion

While the performance gains from hierarchical clustering were modest, the research validates the approach of mimicking human logic in automated grading. The high performance of the phrase-level alignment tool suggests a promising future for integrated systems. By combining high-accuracy alignment with hierarchical classification, future iterations could provide both precise grades and clear, segment-by-segment explanations to help students learn from their mistakes.

Summary updated

Abstract

Lan honetan erantzunen kalifikazio automatikorako lehen urratsak azaltzen ditugu. Urrats horiek azaltzeko irakaskuntzaren domeinuko bi sistema deskribatzen ditugu:bata, erantzunak hierarkikoki antolatzen eta kalifikatzen dituena; eta, bestea, kalifikazioaren azalpena emateko asmoz, erantzunaren zatiak erreferentzia baten aurka alderatzeko gai dena. Horretaz gain, irakaskuntzaren alorreko ataza internazional bat aztertuz alorreko teknika erabilienak mahairatzen ditugu.
BibTeX
@inproceedings{agirre2015erantzunen,
  author = {Eneko Agirre and Itziar Aldabe and Oier López de Lacalle and Iñigo López-Gazpio and Montse Maritxalar},
  title = {{Erantzunen kalifikazio automatikorako lehen urratsak}},
  booktitle = {EKAIA},
  year = {2015},
  doi = {10.1387/ekaia.14530},
}
Figure