Semantic Networks Extracted from Students’ Think-Aloud Data are Correlated with Students’ Learning Performance

Pingjing Yang, Sullam Jeoung, Jennifer G. Cromley, Jana Diesner2025 · DOI

Abstract

When students reflect on their learning from a textbook via think-aloud processes, network representations can be used to capture the concepts and relations from these data. This study brings methods from entity and relation extraction using classic and LLM-based methods to the application domain of educational psychology. We built a ground-truth baseline of relational data that represents relevant textbook-based information as a semantic network. Among the tested models, SPN4RE and LUKE achieved the best performance in extracting concepts and relations from students' verbal data. Network representations of students' verbalizations varied in structure, reflecting different learning processes. Correlating the students' semantic networks with learning outcomes revealed that denser and more interconnected semantic networks were associated with more elaborated knowledge acquisition. Structural features such as the number of edges and surface overlap with textbook networks significantly correlated with students' posttest performance.

Introduction: The Challenge of Analyzing Think-Aloud Data

In educational psychology, "think-aloud" protocols—where students verbalize their thoughts while studying—are considered the gold standard for understanding cognitive processes. However, analyzing these transcripts presents a significant bottleneck, typically requiring extensive manual coding by experts. This study addresses the "messy data" problem inherent in student speech (fragmented sentences and ambiguous pronouns) and the automation gap in converting these verbalizations into structured semantic networks that can predict learning outcomes.

Methodology: A Multi-Stage Extraction Pipeline

The researchers developed a robust pipeline to transform raw audio transcripts into predictive models. The process begins with Data Preprocessing, where Large Language Models (LLMs) perform coreference resolution and sentence completion to "fill in the blanks" of informal student speech.

The multi-stage pipeline for extracting semantic networks from educational data and analyzing their correlation with learning outcomes.

Following preprocessing, the pipeline utilizes two main approaches for Relation Extraction (RE):

  1. Pre-trained Language Models (PLMs): Fine-tuned models like LUKE and SpanBERT for classification, and SPN4RE for joint entity/relation extraction.
  2. Large Language Models (LLMs): Llama 3.1 and GPT-3.5-Turbo using in-context learning strategies.

To handle linguistic variety, 226 raw predicates were mapped to 46 expert-defined "consolidated" stems. Finally, individual student semantic networks were constructed and compared against a textbook-derived "gold standard" using graph metrics and HIMATT framework matching.

Results: Extraction Accuracy and Predictive Power

The study revealed an "Accuracy Paradox." While fine-tuned PLMs like LUKE achieved near-perfect extraction performance (F1 = 0.99) on consolidated relations, LLMs often struggled with raw accuracy. However, the structural properties of networks generated by LLMs frequently showed stronger correlations with student performance.

Elastic Net regression coefficients showing the relationship between network features and student learning performance across different extraction models.

As shown in the regression analysis, structural features such as the number of edges and network density significantly correlated with posttest performance across various model configurations. Denser, more interconnected networks were consistently associated with higher scores, confirming that "elaborated" knowledge—where concepts are woven together—is a hallmark of deep learning.

Discussion and Key Insights

  • Density = Mastery: The correlation between network density and learning outcomes suggests that the global structure of a student's knowledge is a more powerful indicator of mastery than the local accuracy of individual facts.
  • Surface Matching: Overlap between student-mentioned concepts and the textbook's reference network (Surface Matching) emerged as a consistent predictor of success.
  • LLMs as Preprocessors: The use of LLMs to clean conversational data was critical for making downstream extraction models effective on informal speech.

Conclusion

This research demonstrates that automatically extracted semantic networks can effectively predict learning performance. While the study was limited to introductory biology and required initial expert setup for relation consolidation, it paves the way for automated, real-time feedback systems in classrooms and large-scale online courses. Future work will focus on generalizability across domains and developing more accessible toolkits for educators.

Summary updated

Abstract

When students reflect on their learning from a textbook via think-aloud processes, network representations can be used to capture the concepts and relations from these data.What can we learn from the resulting network representations about students' learning processes, knowledge acquisition, and learning outcomes?This study brings methods from entity and relation extraction using classic and LLM-based methods to the application domain of educational psychology.We built a ground-truth baseline of relational data that represents relevant (to educational science), textbook-based information as a semantic network.Among the tested models, SPN4RE and LUKE achieved the best performance in extracting concepts and relations from students' verbal data.Network representations of students' verbalizations varied in structure, reflecting different learning processes.Correlating the students' semantic networks with learning outcomes revealed that denser and more interconnected semantic networks were associated with more elaborated knowledge acquisition.Structural features such as the number of edges and surface overlap with textbook networks significantly correlated with students' posttest performance.
BibTeX
@misc{yang2025semantic,
  author = {Pingjing Yang and Sullam Jeoung and Jennifer G. Cromley and Jana Diesner},
  title = {{Semantic Networks Extracted from Students’ Think-Aloud Data are Correlated with Students’ Learning Performance}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1309},
}
Figure