Abstract: Lan honetan erantzunen kalifikazio automatikorako lehen urratsak azaltzen ditugu. Urrats horiek azaltzeko irakaskuntzaren domeinuko bi sistema deskribatzen ditugu: bata, erantzunak hierarkikoki antolatzen eta kalifikatzen dituena; eta, bestea, kalifikazioaren azalpena emateko asmoz, erantzunaren zatiak erreferentzia baten aurka alderatzeko gai dena. Horretaz gain, irakaskuntzaren alorreko ataza internazional bat aztertuz alorreko teknika erabilienak mahairatzen ditugu.
Introduction
In the field of Educational Technology (EdTech), the development of systems capable of grading open-ended student answers with human-like accuracy is a significant challenge. While multiple-choice assessments are easily automated, they often fail to capture the nuance of a student's conceptual understanding. This research explores Automated Short Answer Grading (ASAG) within a Basque linguistic context, addressing two primary hurdles: the classification of nuanced student errors and the need for explainable feedback.
Methodology
The researchers propose two complementary approaches to move beyond "black box" grading systems:
1. Hierarchical EHU-ALM
Most existing systems treat grading as a flat classification task. However, human logic often groups similar types of errors. The EHU-ALM system was adapted into a hierarchical model using clustering based on a confusion matrix. This approach transforms the problem into a decision tree, allowing the system to first make broad distinctions—such as whether an answer is within the correct domain—before attempting to distinguish between finer categories like "Correct" and "Partially Correct."
2. Phrase-Level Alignment
To improve explainability, a second system was developed to decompose sentences into chunks (phrases). Utilizing the Hungarian Algorithm, the system finds the optimal mapping between student phrases and reference answer phrases. This granular alignment allows the system to identify exactly which components of a student's response are missing or incorrect, providing a foundation for intelligent tutoring.
Evaluation and Results
The systems were evaluated using benchmarks from the SemEval (Semantic Evaluation) workshops, specifically the 2013 Joint Student Response Analysis and the 2015 Semantic Textual Similarity tasks.
In the classification task, the hierarchical version of the EHU-ALM system demonstrated a performance improvement over the standard flat classifier. As shown in the results table below, the hierarchical approach achieved an F-score of 0.426 compared to the baseline of 0.415.
Furthermore, the segment-alignment system performed exceptionally well in the SemEval 2015 task, achieving a score of 0.8846, which significantly outperformed the task average of 0.8193 and nearly matched the top-performing international system.
Conclusion
While the performance gains from hierarchical clustering were modest, the research validates the approach of mimicking human logic in automated grading. The high performance of the phrase-level alignment tool suggests a promising future for integrated systems. By combining high-accuracy alignment with hierarchical classification, future iterations could provide both precise grades and clear, segment-by-segment explanations to help students learn from their mistakes.
Summary updated