Measuring Students' Engagement with Digital Interactive Textbooks by Analyzing Clickstream Data

Breanne CrockettAAAI 2022 · DOI

TL;DR

This paper explores how students interact with digital interactive textbooks by analyzing clickstream data from a chemical engineering platform. The authors identified three distinct student engagement patterns, suggesting that clickstream data can be a valuable tool for understanding student learning beyond simple completion rates and can inform the design of more effective educational technologies.

What Problem Does This Paper Address?

Traditional textbooks have long faced the challenge of students skipping reading assignments, with studies suggesting up to 70% of students don't read assigned material. While digital interactive textbooks offer a more engaging alternative, simply tracking completion rates doesn't reveal if students are truly understanding the content or just clicking through to satisfy requirements. This paper addresses the gap in understanding how students engage with interactive digital content, moving beyond mere completion to analyze the depth of their interaction and mental workload. The goal is to use this understanding to improve student success in educational settings.

Methods

The core methodology involves analyzing clickstream data generated by students interacting with a chemical engineering textbook called ZyBook. This data captures every click a student makes, including initiating animations, replaying steps, and navigating content. The primary analytical technique employed is K-Means clustering with Euclidean distance. This algorithm was used to group students based on their viewing times for each animation within the textbook. The authors experimented with different values of 'k' (the number of clusters) to find the optimal grouping. Beyond clustering, the paper mentions the potential for using this data to inform feature selection for future machine learning models.

Datasets

The study utilized clickstream data from a chemical engineering ZyBook collected over five semesters, encompassing 111 students. This dataset represents a single cohort and is described as "small" by the authors, particularly when considering potential future work involving more extensive data collection.

Key Insights and Ideas

  • Beyond Completion Rates: The paper strongly advocates for moving beyond simple completion metrics in digital learning environments. Clickstream data offers a richer, more nuanced view of student engagement.
  • Identifying Engagement Archetypes: The identification of three distinct student viewing patterns (low, medium, and high engagement) is a significant insight. This suggests that student interaction with digital content is not monolithic.
  • Clickstream as a Proxy for Mental Workload: While acknowledging limitations, the authors propose that viewing time, as captured by clickstream data, can serve as an indicator of the effort students invest in digesting material.
  • Informing Future Design: The identified patterns can directly inform the design of future interactive textbooks and animations, helping educators create content that better engages students and potentially optimizes mental workload for learning.

Key Results

The K-Means clustering analysis consistently revealed three primary clusters of student engagement across various feature sets (overall view time, average time per chapter, average time per animation category). These clusters represent:

  • Cluster 1 (Bottom): Students with consistently shorter view times, likely clicking through for completion points.
  • Cluster 2 (Middle): Students who earnestly watch each animation.
  • Cluster 3 (Top): Students with consistently longer view times, potentially rewatching content or spending more time processing it.

The paper provides a figure illustrating these clusters, showing the centroid of each cluster's view time across animation numbers. While specific quantitative performance metrics (like accuracy of prediction) are not detailed in this summary, the qualitative finding of three distinct engagement patterns is the primary result.

Limitations and Future Work

Limitations: * The current dataset is described as "small," limiting the generalizability of findings to broader student populations or different academic disciplines. * View time, while an indicator of effort, is not a definitive measure of mental workload or actual understanding.

Future Work: * Generative Models: Constructing generative models for student view times to forecast future engagement. * Measuring Mental Workload: Integrating clickstream data with physiological sensor data (e.g., heart rate variability, electrodermal activity) to more accurately measure students' mental workload, validated against self-reported measures like the NASA Task Load Index. * Correlation with Success: Investigating the correlation between student engagement patterns, mental demand, and academic success. * "Ideal" Animation Features: Identifying features of animations that require optimal mental workload for improved student learning. * Broader Data Collection: Expanding data collection to include more cohorts, textbooks, and universities to build more robust machine learning models.

Summary updated

BibTeX
@inproceedings{crockett2022measuring,
  author = {Breanne Crockett},
  title = {{Measuring Students' Engagement with Digital Interactive Textbooks by Analyzing Clickstream Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21702},
}
Figure