Meta Transfer Learning for Early Success Prediction in MOOCs

Vinitra Swamy, Mirko Marras, Tanja Käser

TL;DR

Researchers have developed a "meta transfer learning" framework that predicts whether a student will pass or fail a MOOC early in the semester, even for brand-new courses. By combining student behavior data with course metadata (like topic and difficulty level) using attention-based neural networks, these models can be trained on one set of courses and successfully applied to entirely different ones.

What Problem Does This Paper Address?

Massive Open Online Courses (MOOCs) are famous for high enrollment but notorious for high dropout rates. To help students, educators need to identify who is struggling early enough to intervene.

The catch? Most machine learning models are "lazy"—they are trained on a specific course using data from previous years. This creates two major hurdles:

  1. The Cold Start Problem: How do you predict success for a brand-new course that has never been run before?
  2. The Label Delay: You don't know who passed or failed a current course until it's over, making it impossible to train a "live" model for immediate use.

The authors tackle the challenge of transferability: building a model that learns general patterns of student success across diverse subjects—from Microcontrollers to Digital Signal Processing—and applies them to new contexts.

Methods

The researchers proposed three deep learning architectures based on Bidirectional Long Short-Term Memory (BiLSTM) networks, which are excellent at processing sequences of student actions over time.

  1. Behavior Only (BO): A baseline model using only 45 hand-crafted behavioral features (e.g., how often a student pauses videos, how many quizzes they attempt, and their study regularity).
  2. Behavior + Time-wise Meta (BTM): This model injects course metadata (like the course title and difficulty) into every single week of the student's activity log.
  3. Behavior + Static Meta (BSM): The most sophisticated approach. It uses Attention Mechanisms to weigh which course characteristics (e.g., "Is this a Master's level course?") are most relevant to the behavioral patterns. It processes behavior and metadata in separate branches before merging them into a final prediction.

To handle text-based metadata like course descriptions, they used FastText embeddings to turn syllabus language into mathematical vectors the model could understand.

Datasets

The study utilized a massive dataset of 26 MOOCs hosted by a European university between 2013 and 2015.

  • Scale: 145,714 initial enrollments, filtered down to 73,042 students to focus on those who didn't drop out in the first week (the "harder" group to predict).
  • Diversity: The courses spanned two languages (French and English), three university levels (Bachelor, Master, Propedeutic), and seven distinct topics.
  • Interactions: Millions of fine-grained "clickstream" events, such as play, pause, and quiz submissions.

Key Insights and Ideas

  • Metadata as a Bridge: Course metadata acts as a "translator." If a model knows a new course is a "Master's level Engineering" class, it can look back at similar classes it has seen before to better interpret student behavior.
  • The "Quiz Effect": Courses with more quizzes generally yielded more accurate predictions. Quizzes provide "high-signal" data points that are more discriminative of success than just watching videos.
  • Context Matters More Than History: Surprisingly, models that combined behavior with course metadata often performed better than models trained specifically on previous versions of the same course. This suggests that "meta-information" helps the model generalize better than just seeing more of the same data.
  • Early Signal: The models achieved high accuracy using only 40% of the course duration, providing a significant window for instructors to offer support.

Key Results

  • Transfer Performance: The best transfer models (N-1 Diff) achieved a Balanced Accuracy (BAC) of around 70% on courses the model had never seen during training.
  • Meta-Learning Boost: Adding meta-information improved performance by an average of 11.45% over simple one-course transfer baselines.
  • Feature Importance: For very early predictions (40% mark), the course Level, Title, and Duration were the most critical meta-features. As the course progressed (60% mark), the Long Description became more influential.

Limitations and Future Work

  • Data Aging: The dataset is from 2013–2015. Student behavior and MOOC platforms have evolved significantly since then, especially post-COVID.
  • Single Institution: All data came from one university. Patterns might differ across different cultures or platforms (e.g., Coursera vs. edX).
  • Hand-Crafted Features: The model relies on 45 pre-defined features. The authors suggest that future work could use "End-to-End" learning, where the model learns directly from raw clickstream data without human intervention.
  • Modality: The study focused on videos and quizzes; future models could incorporate forum posts or textbook readings for a more holistic view of student life.

Summary updated

Abstract

Despite the increasing popularity of massive open online courses (MOOCs), many suffer from high dropout and low success rates. Early prediction of student success for targeted intervention is therefore essential to ensure no student is left behind in a course. There exists a large body of research in success prediction for MOOCs, focusing mainly on training models from scratch for individual courses. This setting is impractical in early success prediction as the performance of a student is only known at the end of the course. In this paper, we aim to create early success prediction models that can be transferred between MOOCs from different domains and topics. To do so, we present three novel strategies for transfer: 1) pre-training a model on a large set of diverse courses, 2) leveraging the pre-trained model by including meta information about courses, and 3) fine-tuning the model on previous course iterations. Our experiments on 26 MOOCs with over 145,000 combined enrollments and millions of interactions show that models combining interaction data and course information have comparable or better performance than models which have access to previous iterations of the course. With these models, we aim to effectively enable educators to warm-start their predictions for new and ongoing courses.
BibTeX
@inproceedings{swamy2022meta,
  author = {Vinitra Swamy and Mirko Marras and Tanja Käser},
  title = {{Meta Transfer Learning for Early Success Prediction in MOOCs}},
  booktitle = {L@S},
  year = {2022},
  doi = {10.1145/3491140.3528273},
}

Loading related content...

Figure