Data-driven modelling and characterisation of task completion sequences in online courses

Robert L. Peach, Sam F. Greenbury, Iain G. Johnston, Sophia N. Yaliraki, David Lefevre, Mauricio Barahona

Abstract

Traditional learning analytics often rely on "temporally averaged" feature vectors, such as total time spent or number of logins, which fail to capture the longitudinal dynamics of student progression. This study proposes a novel framework to analyze the specific order in which students complete tasks. By applying Hypercubic Transition Path Sampling (HyperTraPS), a Bayesian model originally used in evolutionary biology, we demonstrate that task sequences are highly predictive of student performance. Our findings reveal that high-performing students exhibit greater "micro-flexibility" within sessions while maintaining "macro-consistency" with the course schedule. This approach enables early identification of at-risk students and provides actionable insights for course design.

Introduction

Most current learning analytics frameworks summarize student behavior into static vectors. While useful for broad classification, this "feature engineering" approach ignores the intrinsic temporal nature of learning. A student who attempts a quiz before engaging with the foundational reading material demonstrates a fundamentally different learning strategy than one who follows the intended sequence.

Modeling these trajectories is computationally challenging; a course with over 100 tasks presents a state space of astronomical proportions ($2^{123}$). This paper addresses this challenge by treating task completions as directed paths and employing probabilistic modeling to identify significant deviations from nominal course designs.

Conceptual comparison of nominal course design versus actual student task sequences.

Methodology

The researchers introduced a Sequence Data Framework to map every task—including videos, readings, quizzes, and discussion posts—as a node in a directed path.

  1. HyperTraPS (Hypercubic Transition Path Sampling): The core of the analysis is a Bayesian model that reduces the state space by assuming the probability of the next task depends on the set of tasks already completed.
  2. Transition Probability Matrices: These were used to visualize the likelihood of transitions between specific tasks, highlighting where student behavior diverged from the "nominal" path intended by instructors.
  3. Classification: A Naïve Bayes Classifier utilized the learned trajectories to categorize students into "high performing" (top 25%) and "low performing" (bottom 25%) cohorts.

Dataset

The study analyzed a 10-week "Corporate Finance" module from an online MBA program. The dataset included 81 post-experience learners navigating 123 unique tasks. High-resolution, time-stamped logs provided a granular view of every transition and completion.

Key Results

Macro-Consistency vs. Micro-Flexibility

A primary insight is the difference in navigational strategies between performance tiers. High-performing students generally adhered to the weekly session structure (macro-consistency) but were more likely to jump between tasks within a single session (micro-flexibility). This suggests a non-linear, active engagement style. In contrast, low performers often followed the nominal order more rigidly or skipped foundational tasks to attempt graded coursework prematurely—a phenomenon termed the "Coursework Skip."

Early Performance Prediction

The Bayesian model demonstrated significant predictive power early in the term. Researchers could distinguish between high and low performers with high certainty after only 30 tasks, representing approximately 25% of the course duration.

Visualizing Behavioral Deviations

The use of heat maps allowed for the identification of "critical junctures" in the curriculum. These are points where the cohort's behavior splits into distinct branches, often coinciding with interactive tasks or complex application-based coursework.

Heatmap showing the relationship between nominal task ordering and actual completion order, highlighting behavioral patterns like bimodality and branching.

Discussion and Conclusion

This research highlights that how students move through a course is as important as what they do. The correlation between sequence deviations and self-reported confidence suggests that students use non-linear paths to "test" their knowledge when feeling uncertain.

While the study is limited by a relatively small sample size (N=81), it provides a scalable method for real-time learning analytics. Future work involves integrating these sequence models into Learning Management Systems (LMS) to provide educators with early-warning alerts based on student trajectories rather than just aggregate scores.

Summary updated

Abstract

The intrinsic temporality of learning demands the adoption of methodologies capable of exploiting time-series information. In this study we leverage the sequence data framework and show how data-driven analysis of temporal sequences of task completion in online courses can be used to characterise personal and group learners' behaviors, and to identify critical tasks and course sessions in a given course design. We also introduce a recently developed probabilistic Bayesian model to learn sequence trajectories of students and predict student performance. The application of our data-driven sequence-based analyses to data from learners undertaking an on-line Business Management course reveals distinct behaviors within the cohort of learners, identifying learners or groups of learners that deviate from the nominal order expected in the course. Using course grades a posteriori, we explore differences in behavior between high and low performing learners. We find that high performing learners follow the progression between weekly sessions more regularly than low performing learners, yet within each weekly session high performing learners are less tied to the nominal task order. We then model the sequences of high and low performance students using the probablistic Bayesian model and show that we can learn engagement behaviors associated with performance. We also show that the data sequence framework can be used for task centric analysis; we identify critical junctures and differences among types of tasks within the course design. We find that non-rote learning tasks, such as interactive tasks or discussion posts, are correlated with higher performance. We discuss the application of such analytical techniques as an aid to course design, intervention, and student supervision.
BibTeX
@inproceedings{peach2021data,
  author = {Robert L. Peach and Sam F. Greenbury and Iain G. Johnston and Sophia N. Yaliraki and David Lefevre and Mauricio Barahona},
  title = {{Data-driven modelling and characterisation of task completion sequences in online courses}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-81709-3},
}
Figure