Understanding learner behaviour in online courses with Bayesian modelling and time series characterisation

Robert L. Peach, Sam F. Greenbury, Iain G. Johnston, Sophia N. Yaliraki, David Lefèvre, Mauricio Barahona

Abstract

The intrinsic temporality of learning demands the adoption of methodologies capable of exploiting time-series information. In this study we leverage the sequence data framework and show how data-driven analysis of temporal sequences of task completion in online courses can be used to characterise personal and group learners' behaviors, and to identify critical tasks and course sessions in a given course design. We also introduce a recently developed probabilistic Bayesian model to learn sequential behaviours of students and predict student performance. The application of our data-driven sequence-based analyses to data from learners undertaking an on-line Business Management course reveals distinct behaviors within the cohort of learners, identifying learners or groups of learners that deviate from the nominal order expected in the course. Using course grades a posteriori, we explore differences in behavior between high and low performing learners. We find that high performing learners follow the progression between weekly sessions more regularly than low performing learners, yet within each weekly session high performing learners are less tied to the nominal task order. We then model the sequences of high and low performance students using the probablistic Bayesian model and show that we can learn engagement behaviors associated with performance. We also show that the data sequence framework can be used for task-centric analysis; we identify critical junctures and differences among types of tasks within the course design. We find that non-rote learning tasks, such as interactive tasks or discussion posts, are correlated with higher performance. We discuss the application of such analytical techniques as an aid to course design, intervention, and student supervision.

Introduction

In online education, success is frequently measured through aggregated metrics such as total hours logged or average quiz scores. However, these "temporally averaged" metrics ignore the longitudinal dynamics of how a student actually navigates a course. This research argues that the sequence of learning tasks is a powerful, underutilized signal for understanding student behavior and predicting academic performance. By analyzing the specific order in which students complete tasks, we can identify "critical junctures" where learners deviate from the intended curriculum and predict final grade groups (top 25% vs. bottom 25%) with high accuracy.

Methodology

The study analyzed data from 81 post-experience learners in a 10-week "Corporate Finance" MBA module. The curriculum consisted of 123 unique tasks, including coursework, quizzes, polls, and discussion posts. The researchers introduced a Sequence Data Framework that treats each student’s journey as an ordered set of task IDs.

Key analytical techniques included:

  • Transition Matrices: Calculating the probability of moving from one specific task to another to visualize "nominal" (intended) versus "actual" behavior.
  • HyperTraPS (Hypercubic Transition Path Sampling): A Bayesian probabilistic model used to handle the massive state space of possible completion sequences ($2^{123}$). This model learns the "basal rate" of task acquisition and how completing one task influences the probability of completing subsequent ones.
Aggregate task completion matrix showing deviations from nominal course order.

As shown in the aggregate matrix above, while the cohort generally follows a diagonal path (the nominal order), significant deviations occur, characterized by branching out of session structures and the emergence of unique task sequence sets as the course progresses.

Results and Discussion

The analysis revealed distinct behavioral archetypes based on academic performance. High performers generally followed the weekly session order (e.g., Session 1 $\rightarrow$ 2 $\rightarrow$ 3) more strictly than low performers. However, within a single week's session, high performers were more likely to deviate from the nominal task order, suggesting a more organic exploration of material or prioritization based on personal needs.

In contrast, low performers—often termed "deadline chasers"—tended to follow the internal order of a session strictly but frequently "jumped" across weeks, bypassing foundational active-learning tasks to reach graded coursework deadlines.

Individual learner paths and cohort-wide distance from nominal order sorted by grade.

Using the Bayesian model, researchers could distinguish between high and low performers with high certainty after the completion of only 30 tasks (approximately 25% of the course). Furthermore, tasks requiring active input, such as discussion posts, were the strongest predictors of success; 19 out of 20 discussion tasks were completed more frequently by the top 25% of students. Self-reported confidence data also correlated with these sequences: students with lower confidence were statistically more likely to skip difficult interactive tasks in favor of quizzes.

Conclusion

This sequence-based approach provides a more granular understanding of student engagement than traditional metrics. While limited by a relatively small sample size in a single MBA module, the findings suggest that monitoring task sequences can identify at-risk students early in a term. Future work will focus on validating these models in larger cohorts, such as MOOCs, and testing real-time interventions to nudge students toward critical active-learning tasks.

Summary updated

Abstract

The intrinsic temporality of learning demands the adoption of methodologies capable of exploiting time-series information. In this study we leverage the sequence data framework and show how data-driven analysis of temporal sequences of task completion in online courses can be used to characterise personal and group learners' behaviors, and to identify critical tasks and course sessions in a given course design. We also introduce a recently developed probabilistic Bayesian model to learn sequential behaviours of students and predict student performance. The application of our data-driven sequence-based analyses to data from learners undertaking an on-line Business Management course reveals distinct behaviors within the cohort of learners, identifying learners or groups of learners that deviate from the nominal order expected in the course. Using course grades a posteriori, we explore differences in behavior between high and low performing learners. We find that high performing learners follow the progression between weekly sessions more regularly than low performing learners, yet within each weekly session high performing learners are less tied to the nominal task order. We then model the sequences of high and low performance students using the probablistic Bayesian model and show that we can learn engagement behaviors associated with performance. We also show that the data sequence framework can be used for task-centric analysis; we identify critical junctures and differences among types of tasks within the course design. We find that non-rote learning tasks, such as interactive tasks or discussion posts, are correlated with higher performance. We discuss the application of such analytical techniques as an aid to course design, intervention, and student supervision.
BibTeX
@inproceedings{peach2021understanding,
  author = {Robert L. Peach and Sam F. Greenbury and Iain G. Johnston and Sophia N. Yaliraki and David Lefèvre and Mauricio Barahona},
  title = {{Understanding learner behaviour in online courses with Bayesian modelling and time series characterisation}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-81709-3},
}
Figure