Personalized Recommendations in EdTech: Evidence from a Randomized Controlled Trial

Keshav Agrawal, Susan Athey, Ayush Kanodia, Georges Vivien HoungbononarXiv 2022 · DOI · arXiv

TL;DR

This paper presents a large-scale randomized controlled trial (RCT) on the "Freadom" educational app, demonstrating that switching from human-curated content to AI-driven personalized recommendations (Matrix Factorization) boosts engagement in the personalized section by 60% and increases overall app usage by 14%. The study highlights that personalization is particularly effective for "niche" users and heavy users, proving that even in structured learning environments, individual preference is a powerful driver of engagement.

What Problem Does This Paper Address?

While personalization is the backbone of giants like Netflix and Spotify, its impact on EdTech—specifically for children—has remained under-studied. There is a common assumption in education that learners (especially children) follow a set routine or the requirements of parents and teachers, making individual "tastes" less relevant.

The authors tackle three main questions:

  1. Does personalization actually increase engagement in a learning app, or do kids just stick to what’s assigned?
  2. Does increased engagement in one section "cannibalize" time from other sections, or does it grow the total time spent learning?
  3. Who benefits most from these algorithms—the casual users or the "power users" with niche interests?

Methods

The researchers partnered with Stones2Milestones to test their "Freadom" app. They moved through a three-stage pipeline:

  • Model Selection: They compared a simple mean model, a Two-Way Fixed Effects (TWFE) popularity-based model, and a Matrix Factorization (MF) collaborative filtering model using historical data. The MF model (implemented in PyTorch) won on Mean Squared Error (MSE).
  • The Metric: They defined a "Story Engagement" score: 1.0 for completion, 0.5 for starting, 0.3 for viewing the description, and 0 for skipping.
  • The RCT: They randomized 7,750 users. The Control Group saw stories curated by human editors (the status quo). The Treatment Group saw stories ranked by the MF algorithm in the "Recommended Story" section.
  • Ranking Logic: To keep things fresh, they implemented a re-ranking routine that removed stories a user had already started or ignored for two consecutive days.

Datasets

The study used internal log data from the Freadom app, an English-learning platform for children aged 3–12 in India.

  • Training/Validation: Historical logs of user-story interactions.
  • RCT Sample: 7,750 users who had interacted with at least 60 stories (to avoid the "cold-start" problem where the algorithm has no data to work with).
  • Content: A library of approximately 2,200 short, illustrated stories.

Key Insights and Ideas

  • No Cannibalization: One of the most important findings is that users didn't just shift their time from the "Trending" section to the "Recommended" section. Instead, the extensive margin increased—users substituted non-app activities (like games or TV) for more time in the educational app.
  • The "Niche" Advantage: Human editors naturally pick "crowd-pleasers." Consequently, users with "niche" tastes (those who prefer stories that aren't globally popular) saw the biggest gains from personalization. The algorithm "found" their interests in a way a human editor never could.
  • Data Density Matters: The magnitude of the benefit scaled with the amount of historical data available. "Heavy users" with long interaction histories received much more accurate recommendations and showed higher engagement boosts than newer users.
  • Discovery via Personalization: Even users who previously ignored the "Recommended" section started using it once it became personalized, suggesting they "scouted" the app and noticed the improved quality.

Key Results

  • Section-Specific Boost: Total Story Engagement in the personalized section increased by 60%.
  • Overall App Growth: Total app usage across all sections increased by 14%.
  • Completion Rates: Treated users completed 78% more stories and spent 87% more time reading in the personalized section.
  • Statistical Significance: The results were robust across different estimators (Difference-in-means, Linear Regression, and AIPW).

Limitations and Future Work

  • Engagement vs. Learning: The authors acknowledge they optimized for engagement, not necessarily pedagogical outcomes. A story a child loves might not be the one that teaches them the most difficult vocabulary. Bridging the gap between "what you want to read" and "what you need to learn" is a major future hurdle.
  • The Cold-Start Problem: The study intentionally excluded new users with fewer than 60 interactions. Matrix Factorization struggles when data is sparse. Future work is needed to see how hybrid models (combining content-based filtering with collaborative filtering) could help new users.
  • Long-term Effects: The RCT lasted two weeks. Whether the "novelty effect" wears off or if personalization leads to long-term literacy gains remains an open question.

Summary updated

Abstract

We study the impact of personalized content recommendations on the usage of an educational app for children. In a randomized controlled trial, we show that the introduction of personalized recommendations increases the consumption of content in the personalized section of the app by approximately 60%. We further show that the overall app usage increases by 14%, compared to the baseline system where human content editors select stories for all students at a given grade level. The magnitude of individual gains from personalized content increases with the amount of data available about a student and with preferences for niche content: heavy users with long histories of content interactions who prefer niche content benefit more than infrequent, newer users who like popular content. To facilitate the move to personalized recommendation systems from a simpler system, we describe how we make important design decisions, such as comparing alternative models using offline metrics and choosing the right target audience.
BibTeX
@misc{agrawal2022personalized,
  author = {Keshav Agrawal and Susan Athey and Ayush Kanodia and Georges Vivien Houngbonon},
  title = {{Personalized Recommendations in EdTech: Evidence from a Randomized Controlled Trial}},
  year = {2022},
  eprint = {2208.13940},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.13940},
}
Figure