Predicting MOOC Dropout over Weeks Using Machine Learning Methods

Marius Kloft, Felix Stiehler, Zhilin Zheng, Niels Pinkwart2014 · DOI

Abstract

With high dropout rates as observed in many current larger-scale online courses, mechanisms that are able to predict student dropout become increasingly important. While this problem is partially solved for students that are active in online forums, this is not yet the case for the more general student population. In this paper, we present an approach that works on click-stream data. Among other features, the machine learning algorithm takes the weekly history of student data into account and thus is able to notice changes in student behavior over time. In the later phases of a course (i.e., once such history data is available), this approach is able to predict dropout significantly better than baseline methods.

Introduction: The Retention Crisis in MOOCs

Massive Open Online Courses (MOOCs) represent a significant shift in democratized education, yet they are plagued by a persistent retention crisis. Completion rates often fall below 13%, with dropout rates exceeding 87% in many instances. While previous research has successfully utilized sentiment analysis of discussion forums to predict attrition, these models fail to account for the "silent 90%"—the vast majority of students who never post on forums. This paper addresses the need for a dropout predictor that encompasses the entire student population by analyzing "clickstream" data: the digital breadcrumbs left by page views and video interactions.

Methodology: Temporal SVM and Clickstream Analysis

The proposed approach utilizes Support Vector Machines (SVMs), specifically the LIBLINEAR library, to process 22 numerical features extracted from raw web logs. These features include session counts, active days, and specific video engagement metrics like "skip-ahead" or "re-listen" events.

A key innovation of this method is its temporal awareness. Rather than treating each week in isolation, the model for any given week $i$ incorporates a "history" feature vector encompassing data from all preceding weeks (1 through $i$). This allows the SVM to detect subtle shifts in engagement over time. To handle the highly skewed nature of clickstream data, a logarithmic transformation was applied to normalize distributions before training. Feature importance was further refined using Fisher scores to identify the most predictive behaviors at different course stages.

Dataset Profile: Analyzing the "Silent 90%"

The study utilized a dataset from a psychology MOOC (EMNLP 2014 shared task) involving 20,828 participants and over 3.4 million web log records. The course spanned 12 weeks of lectures, with data collected over a 19-week lifecycle to capture pre- and post-course activity.

Overview of course activity and dropout metrics over the 19-week lifecycle.

As shown in the data overview, the number of active users declines steadily, while the dropout probability exhibits distinct peaks. The initial surge in new users (Week 1) is followed by a sharp drop-off, highlighting the "cold start" challenge in student retention.

Key Insights and Results

The research yielded several critical insights into student behavior and model performance:

  1. The "Silent" Signal: Clickstream data provides a robust signal for the 90-95% of students who do not participate in forums, proving that forum activity is not a prerequisite for accurate dropout prediction.
  2. Evolving Predictors: The features that predict dropout change as the course progresses. Early on, "balanced" engagement (e.g., consistency in active days) is key. By the final weeks, the raw volume of requests becomes the dominant indicator.
  3. Non-Linear Dropout Probability: Dropout probability is not constant. It peaks sharply in the first two weeks and again during the final exam period (Weeks 11-12).
Dropout probability over time, showing peaks at the beginning and end of the course.
  1. Performance Gains: The SVM model's accuracy improves significantly as more historical data becomes available. In the later phases of the course, the model outperformed the trivial baseline by up to 15%. From Week 9 onwards, the predictive power increases substantially as the "history" features provide a clearer picture of student persistence.

Conclusion and Future Directions

This study demonstrates that incorporating temporal history into clickstream-based machine learning models significantly enhances dropout prediction for the general MOOC population. While the current model intentionally excludes forum data to prove the efficacy of clickstream analysis, future work could explore hybrid models using Multiple Kernel Learning to combine both data sources. Additionally, moving from prediction to intervention remains a vital next step, enabling instructors to use these "at-risk" flags to proactively support struggling students.

Summary updated

Abstract

With high dropout rates as observed in many current larger-scale online courses, mechanisms that are able to predict student dropout become increasingly important. While this problem is partially solved for students that are active in online forums, this is not yet the case for the more general student population. In this paper, we present an approach that works on click-stream data. Among other features, the machine learning algorithm takes the weekly history of student data into account and thus is able to notice changes in student behavior over time. In the later phases of a course (i.e., once such history data is available), this approach is able to predict dropout significantly better than baseline methods.
BibTeX
@misc{kloft2014predicting,
  author = {Marius Kloft and Felix Stiehler and Zhilin Zheng and Niels Pinkwart},
  title = {{Predicting MOOC Dropout over Weeks Using Machine Learning Methods}},
  year = {2014},
  doi = {10.3115/v1/w14-4111},
}

Loading related content...

Figure