Abstract
When students reflect on their learning from a textbook via think-aloud processes, network representations can be used to capture the concepts and relations from these data. This study brings methods from entity and relation extraction using classic and LLM-based methods to the application domain of educational psychology. We built a ground-truth baseline of relational data that represents relevant textbook-based information as a semantic network. Among the tested models, SPN4RE and LUKE achieved the best performance in extracting concepts and relations from students' verbal data. Network representations of students' verbalizations varied in structure, reflecting different learning processes. Correlating the students' semantic networks with learning outcomes revealed that denser and more interconnected semantic networks were associated with more elaborated knowledge acquisition. Structural features such as the number of edges and surface overlap with textbook networks significantly correlated with students' posttest performance.
Introduction: The Challenge of Analyzing Think-Aloud Data
In educational psychology, "think-aloud" protocols—where students verbalize their thoughts while studying—are considered the gold standard for understanding cognitive processes. However, analyzing these transcripts presents a significant bottleneck, typically requiring extensive manual coding by experts. This study addresses the "messy data" problem inherent in student speech (fragmented sentences and ambiguous pronouns) and the automation gap in converting these verbalizations into structured semantic networks that can predict learning outcomes.
Methodology: A Multi-Stage Extraction Pipeline
The researchers developed a robust pipeline to transform raw audio transcripts into predictive models. The process begins with Data Preprocessing, where Large Language Models (LLMs) perform coreference resolution and sentence completion to "fill in the blanks" of informal student speech.
Following preprocessing, the pipeline utilizes two main approaches for Relation Extraction (RE):
- Pre-trained Language Models (PLMs): Fine-tuned models like LUKE and SpanBERT for classification, and SPN4RE for joint entity/relation extraction.
- Large Language Models (LLMs): Llama 3.1 and GPT-3.5-Turbo using in-context learning strategies.
To handle linguistic variety, 226 raw predicates were mapped to 46 expert-defined "consolidated" stems. Finally, individual student semantic networks were constructed and compared against a textbook-derived "gold standard" using graph metrics and HIMATT framework matching.
Results: Extraction Accuracy and Predictive Power
The study revealed an "Accuracy Paradox." While fine-tuned PLMs like LUKE achieved near-perfect extraction performance (F1 = 0.99) on consolidated relations, LLMs often struggled with raw accuracy. However, the structural properties of networks generated by LLMs frequently showed stronger correlations with student performance.
As shown in the regression analysis, structural features such as the number of edges and network density significantly correlated with posttest performance across various model configurations. Denser, more interconnected networks were consistently associated with higher scores, confirming that "elaborated" knowledge—where concepts are woven together—is a hallmark of deep learning.
Discussion and Key Insights
- Density = Mastery: The correlation between network density and learning outcomes suggests that the global structure of a student's knowledge is a more powerful indicator of mastery than the local accuracy of individual facts.
- Surface Matching: Overlap between student-mentioned concepts and the textbook's reference network (Surface Matching) emerged as a consistent predictor of success.
- LLMs as Preprocessors: The use of LLMs to clean conversational data was critical for making downstream extraction models effective on informal speech.
Conclusion
This research demonstrates that automatically extracted semantic networks can effectively predict learning performance. While the study was limited to introductory biology and required initial expert setup for relation consolidation, it paves the way for automated, real-time feedback systems in classrooms and large-scale online courses. Future work will focus on generalizability across domains and developing more accessible toolkits for educators.
Summary updated