Digital interventions and habit formation in educational technology

Keshav Agrawal, Susan Athey, Ayush Kanodia, Georges Vivien HoungbononarXiv 2023 · DOI · arXiv

TL;DR

Can the "addictive" design patterns of social media be used for good in education? This study demonstrates that a 100-day gamified reading contest not only boosted immediate engagement by 45% but also created lasting habits, with treated students reading 75% more stories than the control group three months after all incentives were removed.

What Problem Does This Paper Address?

Educational technology (EdTech) faces a persistent "engagement gap." While digital tools offer unprecedented access to learning, they often suffer from high dropout rates and dwindling student motivation. In contrast, entertainment and social media apps excel at habit formation, sometimes to the point of digital addiction.

A major concern for educators is the "crowding out" effect: the fear that providing external rewards (like prizes or leaderboards) will suppress a child's intrinsic motivation to learn, causing them to stop the activity entirely once the reward is gone. This paper investigates whether a contest-based intervention can successfully instill long-term reading habits without undermining intrinsic interest.

Methods

The researchers conducted a large-scale Randomized Controlled Trial (RCT) involving approximately 10,000 children (Grades 1–6) across 138 schools in India.

  • The Platform: Freadom, an app developed by Stones2Milestones that provides short, illustrated English stories.
  • The Intervention: A "100-day reading contest." Students in the treatment group were ranked on a prominent digital leaderboard. The top 1,000 performers were awarded a physical set of six books.
  • The Control: Students had access to a standard, less prominent leaderboard based on various activities but with no rewards for ranking.
  • Experimental Design: Randomization was stratified by school size, grade size, and city. The study tracked three phases: Pre-treatment, Treatment (the contest), and Post-treatment (up to 24 weeks after the contest ended).
  • Metrics: The authors used "Total Stories Completed" and "Total Story Engagement" (a weighted score: 0.3 for viewing a description, 0.5 for starting, and 1.0 for completion). They employed Poisson regression to model story difficulty and Cox Proportional Hazard models to measure student retention.

Datasets

The study utilized proprietary interaction data from the Freadom app. This included:

  • User Interaction Logs: Timestamps for app launches, story selections, and completion rates.
  • Content Metadata: A catalog of thousands of stories labeled by difficulty level (grade-appropriate vs. challenging).
  • Demographic/School Data: School fees (used as a proxy for socioeconomic status), school size, and geographic location.

Key Insights and Ideas

  • Habit Persistence: The most significant finding is that the intervention didn't just work during the contest; it changed behavior long-term. The 75% increase in post-contest usage suggests that the "crowding out" effect did not occur. Instead, the contest acted as a "nudge" that helped students overcome the initial friction of starting a new habit.
  • Intensity over Retention: While the intervention improved 12-week retention by 6% (1.3 percentage points), the primary driver of the results was the intensity of usage among those who stayed. Treated students who remained active used the app much more heavily than active control students.
  • Quality of Engagement: Treated students didn't just "click through" easy content to win. They completed significantly more "difficult" stories (those above their grade level), suggesting the intervention encouraged actual skill progression and learning.
  • Socioeconomic Heterogeneity: The intervention was more effective for students in less expensive schools and for younger students, suggesting that low-cost digital gamification could be a powerful tool for closing educational equity gaps.

Key Results

  • During the Contest: Treated students read 0.96 more stories (a 45% increase) and had nearly double the leaderboard visits compared to the control group.
  • Post-Contest (12 Weeks): Even with no prizes on the line, the treated group read 0.30 more stories per week—a 75% increase over the control group's mean of 0.40.
  • Retention: Treated students were 6% more likely to continue using the app 12 weeks after the contest.
  • Learning Proxy: Treated students completed an average of 0.54 more difficult stories than the control group during the post-contest period.

Limitations and Future Work

  • Learning Outcomes: While story completion and difficulty are strong proxies, the study does not directly measure literacy gains through standardized testing. The authors acknowledge that more research is needed to confirm if increased engagement translates to tangible educational outcomes.
  • Prize Logistics: While the digital intervention is highly scalable, the physical reward (shipping books to 1,000 winners) introduces logistical costs. Future studies could test if purely digital rewards (badges, avatars) achieve similar habit-forming results.
  • Long-term Decay: While the effects persisted for 24 weeks, it remains unknown if these habits last for years or if periodic "re-engagement" contests are necessary to maintain the behavior.

Summary updated

Abstract

As online educational technology products have become increasingly prevalent, rich evidence indicates that learners often find it challenging to establish regular learning habits and complete their programs. Concurrently, online products geared towards entertainment and social interactions are sometimes so effective in increasing user engagement and creating frequent usage habits that they inadvertently lead to digital addiction, especially among youth. In this project, we carry out a contest-based intervention, common in the entertainment context, on an educational app for Indian children learning English. Approximately ten thousand randomly selected learners entered a 100-day reading contest. They would win a set of physical books if they ranked sufficiently high on a leaderboard based on the amount of educational content consumed. Twelve weeks after the end of the contest, when the treatment group had no additional incentives to use the app, they continued their engagement with it at a rate 75\% higher than the control group, indicating a successful formation of a reading habit. In addition, we observed a 6\% increase in retention within the treatment group. These results underscore the potential of digital interventions in fostering positive engagement habits with educational technology products, ultimately enhancing users' long-term learning outcomes.
BibTeX
@misc{agrawal2023digital,
  author = {Keshav Agrawal and Susan Athey and Ayush Kanodia and Georges Vivien Houngbonon},
  title = {{Digital interventions and habit formation in educational technology}},
  year = {2023},
  eprint = {2310.10850},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10850},
}
Figure