Markov Decision Process for MOOC users behavioral inference

Firas Jarboui, Célya Gruson-Daniel, Alain Durmus, Vincent Rocchisani, Sophie-Helene Goulet Ebongue, Anneliese Depoux, Wilfried Kirschenmann, Vianney Perchet

Abstract

Studies on massive open online courses (MOOCs) users discuss the existence of typical profiles and their impact on the learning process of the students. However, defining typical behaviors as well as classifying users accordingly is a difficult task. In this paper, we suggest two methods to model MOOC users' behavior given their log data. We mold their behavior into a Markov Decision Process (MDP) framework, associating user intentions with the MDP reward to facilitate more accurate classification.

Introduction

Understanding how students interact with Massive Open Online Courses (MOOCs) is essential for optimizing e-learning environments. However, accurately classifying user behavior remains a significant challenge in educational data science. Traditional methods often rely on aggregate metrics—such as total clicks or time spent on a page—or rigid, predefined typologies (e.g., "active" vs. "passive" learners). While these metrics provide a surface-level view, they fail to capture the underlying intent of the student.

This research addresses this gap by mathematically modeling user motivation. By framing platform interactions as a Markov Decision Process (MDP), the authors hypothesize that the "rewards" a user seeks to maximize can serve as a robust proxy for their learning goals and behavioral profiles.

Methodology

To model the MOOC environment, the authors define a state-action space where states represent the various pages a user can access (including a logged-out state) and actions represent specific interactions, such as playing a video or attempting a quiz.

Because a user's true goals are latent, the researchers employ Inverse Reinforcement Learning (IRL). This technique works backward from observed log data to infer the reward functions that would logically drive a user's observed policy. Two distinct methodologies were developed to apply this framework:

1. Static Behavior Clustering (SBC)

The SBC approach assumes that a user follows a single, consistent policy throughout the duration of a course. Using Bayesian IRL, the system extracts a reward parameter vector for each user. These vectors are then processed using Label Propagation, a semi-supervised graph algorithm. By starting with a small subset of users manually labeled by experts, the algorithm clusters the entire population based on their inferred reward structures.

2. Dynamic Behavior Clustering (DBC)

Recognizing that student motivations often shift, the DBC approach assumes users transition between a small set of fundamental behavioral modes over time. This is modeled as a switched MDP (sMDP)—effectively a Hidden Markov Model where the hidden states are different MDP policies. The authors utilize a Markov Chain Monte Carlo (MCMC) approach combined with the Viterbi algorithm to determine when and why users switch between these latent modes.

Key Insights: Intent as a Reward

The primary innovation of this work is the shift from raw behavioral metrics to inferred reward parameters. This approach offers several advantages:

  • Interpretability: Clusters are grounded in optimization goals (what the user finds valuable) rather than just activity volume.
  • Decomposition of Complex Profiles: The authors found that complex static profiles (such as a "Late Quitter") are not necessarily unique behaviors, but rather temporal sequences of simpler, fundamental modes (e.g., moving from Learning to Exploration before eventually exiting).
  • Expert Integration: The SBC model benefits from a human-in-the-loop process, where expert review of cluster boundaries helps refine the mathematical definitions of user types.

Results and Behavioral Profiles

The study utilized proprietary log data from the #MOOCLive research project. The models successfully identified distinct behavioral archetypes:

Static Profiles (SBC): The model converged on six expert-approved classifications:

  • Participant: Engages with all course elements.
  • Collaborative: Primarily active in forums and social interactions.
  • Targeting: Focuses exclusively on specific chapters or topics.
  • Auditor: Consumes content (reading/watching) but avoids assessments.
  • Clicker: Skims through content rapidly without deep engagement.
  • Big Starter / Late Quitter: Users who drop off early or late in the course cycle, respectively.

Dynamic Modes (DBC): The dynamic model distilled behavior into three fundamental latent modes:

  1. Exploration: Non-linear skipping through various MOOC sections.
  2. Learning: Sustained, sequential attention to course content.
  3. Certification: A narrow focus on fulfilling requirements, such as quizzes and assignments.

Limitations and Future Directions

While the MDP framework provides a powerful lens for behavioral inference, the authors acknowledge several limitations. The SBC model's assumption of a single policy is often too rigid for long-term courses. Conversely, the DBC model currently bases transitions on state changes (page visits) rather than temporal factors like fatigue.

Most significantly, the authors note that learning is inherently non-Markovian. In a standard MDP, rewards depend only on the current state; however, in education, the reward of consuming content diminishes with repetition. Future work will focus on incorporating non-Markovian reward structures to better reflect the diminishing returns and cumulative nature of the learning process.

Summary updated

Abstract

Studies on massive open online courses (MOOCs) users discuss the existence of typical profiles and their impact on the learning process of the students. However defining the typical behaviors as well as classifying the users accordingly is a difficult task. In this paper we suggest two methods to model MOOC users behaviour given their log data. We mold their behavior into a Markov Decision Process framework. We associate the user's intentions with the MDP reward and argue that this allows us to classify them.
BibTeX
@inproceedings{jarboui2019markov,
  author = {Firas Jarboui and Célya Gruson-Daniel and Alain Durmus and Vincent Rocchisani and Sophie-Helene Goulet Ebongue and Anneliese Depoux and Wilfried Kirschenmann and Vianney Perchet},
  title = {{Markov Decision Process for MOOC users behavioral inference}},
  booktitle = {EMOOCs},
  year = {2019},
  url = {https://openreview.net/forum?id=HbENdQ_o1-c},
}
Figure