Markov Decision Process for MOOC Users Behavioral Inference

Firas Jarboui, Célya Gruson-Daniel, Alain Durmus, Vincent Rocchisani, Sophie-Helene Goulet Ebongue, Anneliese Depoux, Wilfried Kirschenmann, Vianney Perchet

TL;DR

This paper models the behavior of Massive Open Online Course (MOOC) users by framing their platform interactions as a Markov Decision Process (MDP). By using Inverse Reinforcement Learning (IRL) to infer the hidden "rewards" users are trying to maximize, the authors successfully cluster students into interpretable behavioral profiles, offering both a static model for overall course behavior and a dynamic model that captures how users switch strategies over time.

What Problem Does This Paper Address?

Understanding how students interact with MOOCs is crucial for improving e-learning platforms, but accurately classifying user behavior is notoriously difficult. Historically, educational data science has relied on simple aggregate metrics (e.g., total clicks, total time) or rigid, predefined typologies (e.g., "active" vs. "passive" learners). While some recent methods use Hidden Markov Models to track state transitions, simply observing what a user does lacks interpretability regarding why they are doing it.

The authors address this gap by attempting to mathematically model user intent. They hypothesize that if you can figure out what a user perceives as a "reward" on the platform, you can accurately classify their overarching learning behavior and motivations.

Methods

The authors map the MOOC environment to an MDP framework: states are the pages a user can access (plus a logged-out state), and actions are interactions like playing a video or answering a quiz. Because the users' actual goals are unknown, the authors use Inverse Reinforcement Learning (IRL) to work backward from the log data to infer the reward functions driving the users' policies.

They propose two distinct methodologies:

  1. Static Behavior Clustering (SBC): This assumes a user follows a single, unique policy throughout the entire course. The system uses Bayesian IRL to extract a reward parameter vector for each user. Then, using a semi-supervised graph algorithm called Label Propagation, it clusters all users based on a small subset of users that human experts have manually labeled.
  2. Dynamic Behavior Clustering (DBC): Recognizing that users change their minds, this approach assumes users switch between a small set of fundamental behaviors over time. This is modeled as a switched MDP (sMDP)—essentially a Hidden Markov Model where the hidden states are different MDP policies. They use a Markov Chain Monte Carlo (MCMC) approach combined with the Viterbi algorithm to infer when users transition between these latent behavioral modes.

Datasets

The experiments were conducted using proprietary log data from MOOCs published under the #MOOCLive research project, led by the Centre Virchow-Villermé for Public Health. The dataset consists of user interaction logs (pages visited, actions taken, timestamps). No standard open-source benchmarks were utilized.

Key Insights and Ideas

  • Intent as a Mathematical Reward: The core "aha" moment of the paper is shifting the clustering feature space from raw behavioral metrics (clicks, time) to inferred reward parameters. This makes the resulting clusters highly interpretable because they are grounded in optimization goals.
  • Complex Profiles are Just Sequences of Simple Modes: The authors realized that complex, static user profiles (like a "Late Quitter") aren't actually distinct monolithic behaviors. Instead, they are temporal sequences of a few very simple, dynamic modes (e.g., a Late Quitter is just someone who oscillates between Exploration and Learning, but eventually stops).
  • Human-in-the-Loop Refinement: The SBC method utilized an iterative process where the algorithm clustered users, experts reviewed the boundaries, and the definitions were refined. This shows the value of combining unsupervised feature extraction (IRL) with expert domain knowledge.

Key Results

  • Static Profiles (SBC): The static model successfully converged on 6 distinct, expert-approved user classifications: Participant (does everything), Collaborative (highly active on forums), Targeting (only does specific chapters), Auditor (reads but doesn't test), Clicker (skims rapidly), Big Starter (drops off early), and Late Quitter (drops off late).
  • Dynamic Modes (DBC): The dynamic model successfully distilled user behavior into just 3 fundamental latent modes:
  • Exploration: Randomly skipping through the MOOC.
  • Learning: Paying sustained attention to page content.
  • Certification: Focusing strictly on fulfilling course requirements (quizzes/assignments).
  • Visualizations of the DBC model effectively showed how individual users temporally oscillate between these three states (e.g., exploring a bit, settling into learning, trying to certify, and then leaving).

Limitations and Future Work

The authors are highly transparent about the limitations of their models:

  • Rigidity of SBC: The assumption that a user maintains a single policy for an entire course is unrealistic and requires overly specific class definitions to capture nuances.
  • Flawed Mode Switching in DBC: While DBC allows switching, the transitions are based on state changes (visiting different pages) rather than time spent, which doesn't perfectly reflect human learning fatigue or shifts in attention.
  • The Non-Markovian Reality of Learning (Future Work): The biggest acknowledged limitation is that learning is inherently non-Markovian. In an MDP, the reward depends only on the current state. But in a MOOC, reading a chapter for the first time is highly rewarding (you learn something new), while reading it for the 10th time yields diminishing returns. The authors suggest that tackling non-Markovian reward structures is the most vital next step for this research.

Summary updated

Abstract

Studies on massive open online courses (MOOCs) users discuss the existence of typical profiles and their impact on the learning process of the students. However defining the typical behaviors as well as classifying the users accordingly is a difficult task. In this paper we suggest two methods to model MOOC users behaviour given their log data. We mold their behavior into a Markov Decision Process framework. We associate the user’s intentions with the MDP reward and argue that this allows us to classify them.
BibTeX
@inproceedings{jarboui2019markov,
  author = {Firas Jarboui and Célya Gruson-Daniel and Alain Durmus and Vincent Rocchisani and Sophie-Helene Goulet Ebongue and Anneliese Depoux and Wilfried Kirschenmann and Vianney Perchet},
  title = {{Markov Decision Process for MOOC Users Behavioral Inference}},
  booktitle = {EMOOCs},
  year = {2019},
  url = {https://openreview.net/forum?id=HbENdQ_o1-c},
}

Loading related content...

Figure