Power to the Learner: Towards Human-Intuitive and Integrative Recommendations with Open Educational Resources

Sahan Bulathwela, María Pérez‐Ortiz, Emine Yılmaz, John Shawe‐TaylorSustainability 2022 · DOI

TL;DR

This paper introduces a novel educational recommender system, "TrueLearn," that goes beyond traditional approaches by integrating learner interests, knowledge, content novelty, and popularity. It aims to provide more personalized and engaging learning experiences, particularly in the context of open educational resources, and addresses the critical "cold-start" problem in recommendations.

What Problem Does This Paper Address?

The paper tackles the challenge of creating effective educational recommender systems (EdRecSys) that can personalize learning at scale. While e-commerce and entertainment have seen significant advancements in recommendation technology, education has lagged. This gap is problematic because personalized learning is crucial for achieving global educational goals (like SDG 4) and empowering individuals. Existing EdRecSys often struggle to capture the multifaceted nature of learner engagement, which is influenced by not just what a learner knows or is interested in, but also by the novelty of content and its general popularity. Furthermore, a significant hurdle is the "cold-start" problem: how to provide good recommendations when a user has very little interaction history.

Methods

The authors propose a suite of models that build upon each other:

  • TrueLearn Interest: A probabilistic graphical model designed to infer a learner's interest in specific topics based on their interactions with educational content. It uses a Gaussian-based approach to model interest levels and uncertainty.
  • TrueLearn Novel: This model, adapted from prior work, focuses on predicting engagement based on a learner's knowledge state and the novelty of the content.
  • TrueLearn INK (Interest, Novelty, Knowledge): This is an integrative model that combines the predictions from TrueLearn Interest and TrueLearn Novel. They explore two main methods for combining these predictions: probabilistic combinations (AND/OR operators) and meta-learners (like logistic regression and a custom meta-TrueLearn model).
  • TrueLearn PINK (Popularity, Interest, Novelty, Knowledge): This model addresses the cold-start problem by incorporating a population-based engagement predictor as a prior. It uses two hybrid approaches: "switching" (transitioning from population-based to personalized recommendations after a certain number of events) and "stacking" (using a meta-learner to fuse population-based and personalized predictions).

A key aspect across these models is the emphasis on human-intuitive representations and scalability, achieved through techniques like entity linking (using Wikipedia concepts) and online learning.

Datasets

The experiments were conducted using two publicly available datasets drawn from the VideoLectures.Net repository:

  • PEEK dataset: This is the primary dataset used for final evaluation. It contains in-the-wild interaction data from over 20,000 learners consuming fragments of educational video lectures. It's annotated with Wikipedia concepts and is noted for capturing the problem setting well.
  • VLE dataset: This dataset was used to train the context-agnostic engagement predictor for the TrueLearn PINK model.

Key Insights and Ideas

  • Holistic Engagement Prediction: The most significant insight is that combining multiple factors (interest, knowledge, novelty, popularity) leads to substantially better engagement predictions than considering them in isolation.
  • Meta-Learners Shine: For combining different predictive models (like interest and knowledge/novelty), meta-learner approaches generally outperformed simple probabilistic combinations, suggesting that learning how to weigh different signals is more effective.
  • Population Prior for Cold-Start: Incorporating a population-based engagement prior is a highly effective strategy for mitigating the cold-start problem, significantly improving early-stage recommendations.
  • Human-Intuitive Representation is Key: The paper strongly advocates for transparent, human-readable representations (like Wikipedia topics) as crucial for building trust and giving learners agency over their learning pathways.
  • Data Efficiency Matters: The proposed online learning models are designed to be data-efficient, learning rapidly with minimal user interaction, which is vital for lifelong learning scenarios.

Key Results

  • TrueLearn Interest significantly outperformed baseline interest models in terms of recall and F1 score, and was competitive in accuracy.
  • TrueLearn INK models, particularly those using meta-learners (like meta-TrueLearn), showed significant improvements in accuracy, precision, and F1 score compared to individual models and baselines. For instance, meta-TrueLearn achieved the highest performance across these metrics.
  • TrueLearn PINK (both switching and meta-learner versions) demonstrated substantial improvements in accuracy and precision, especially in the early stages of a user's session, effectively addressing the cold-start problem. The meta-learner version of TrueLearn PINK showed superior performance across events.
  • The models showed rapid learning, with performance boosting significantly within the first few events.

Limitations and Future Work

  • Engagement vs. Learning Gains: The paper acknowledges that while TrueLearn excels at predicting engagement (e.g., watch time), it doesn't directly measure actual knowledge acquisition or learning gains. This is a critical objective for educational systems.
  • Independence Assumption: Current models assume that Knowledge Components (KCs) are independent, which is a simplification given that Wikipedia-based KCs are often semantically related.
  • Limited Interaction Signals: The models primarily rely on watch time and don't yet incorporate other valuable interaction signals like pausing, replaying, or skipping.
  • Non-Video OERs: The current framework is focused on video content and needs to be extended to handle other Open Educational Resources (OERs) like PDFs and interactive materials.
  • Future Work: The authors suggest incorporating semantic relatedness between KCs, exploring interest dynamics and forgetting, developing more collaborative AI interfaces, and integrating these models into OER-based learning platforms for larger-scale studies.

Summary updated

Abstract

Educational recommenders have received much less attention in comparison with e-commerce- and entertainment-related recommenders, even though efficient intelligent tutors could have potential to improve learning gains and enable advances in education that are essential to achieving the world’s sustainability agenda. Through this work, we make foundational advances towards building a state-aware, integrative educational recommender. The proposed recommender accounts for the learners’ interests and knowledge at the same time as content novelty and popularity, with the end goal of improving predictions of learner engagement in a lifelong-learning educational video platform. Towards achieving this goal, we (i) formulate and evaluate multiple probabilistic graphical models to capture learner interest; (ii) identify and experiment with multiple probabilistic and ensemble approaches to combine interest, novelty, and knowledge representations together; and (iii) identify and experiment with different hybrid recommender approaches to fuse population-based engagement prediction to address the cold-start problem, i.e., the scarcity of data in the early stages of a user session, a common challenge in recommendation systems. Our experiments with an in-the-wild interaction dataset of more than 20,000 learners show clear performance advantages by integrating content popularity, learner interest, novelty, and knowledge aspects in an informational recommender system, while preserving scalability. Our recommendation system integrates a human-intuitive representation at its core, and we argue that this transparency will prove important in efforts to give agency to the learner in interacting, collaborating, and governing their own educational algorithms.
BibTeX
@inproceedings{bulathwela2022power,
  author = {Sahan Bulathwela and María Pérez‐Ortiz and Emine Yılmaz and John Shawe‐Taylor},
  title = {{Power to the Learner: Towards Human-Intuitive and Integrative Recommendations with Open Educational Resources}},
  booktitle = {Sustainability},
  year = {2022},
  doi = {10.3390/su141811682},
}
Figure