Harnessing Natural Experiments to Quantify the Causal Effect of Badges

Tomasz Kusmierczyk, Manuel Gomez-RodriguezarXiv 2017 · OpenReview

Abstract

A wide variety of online platforms use digital badges to encourage users to take certain types of desirable actions. However, despite their growing popularity, their causal effect on users' behavior is not well understood. This is partly due to the lack of counterfactual data and the myriad of complex factors that influence users' behavior over time. As a consequence, their design and deployment lacks general principles. In this paper, we focus on first-time badges, which are awarded after a user takes a particular type of action for the first time, and study their causal effect by harnessing the delayed introduction of several badges in a popular Q&A website. In doing so, we introduce a novel causal inference framework for badges whose main technical innovations are a robust survival-based hypothesis testing procedure, which controls for the utility heterogeneity across users, and a bootstrap difference-in-differences method, which controls for the random fluctuations in users' behavior over time. We find that first-time badges steer users' behavior if the utility a user obtains from taking the corresponding action is sufficiently low, otherwise, the badge does not have a significant effect. Moreover, for badges that successfully steered user behavior, we perform a counterfactual analysis and show that they significantly improved the functioning of the site at a community level.

Introduction

Digital badges are a staple of modern gamification, utilized by platforms ranging from Duolingo to GitHub to incentivize specific user behaviors. Despite their ubiquity, determining whether these badges actually cause behavioral changes remains a significant challenge. The core issue is one of causal inference: does a badge motivate a user to perform an action, or does it simply reward a user who was already predisposed to that action?

To address this, the authors leverage "natural experiments" found within the history of Stack Overflow. By analyzing badges that were introduced long after the site's inception, they establish a built-in control group of users who performed actions before any reward existed. This allows for a rigorous quantification of the badge's causal impact by comparing behavior before and after the introduction.

Methodology

The study introduces a novel causal inference framework tailored for "first-time badges"—those awarded the first time a user completes a specific task. The framework integrates three key components:

  1. Survival Analysis: The model tracks the "intensity" or rate at which users perform an action for the first time, allowing for changes in this rate following badge introduction.
  2. Robust Hypothesis Testing: To account for "utility heterogeneity"—the fact that different users find different tasks more or less rewarding—the authors use a Gamma distribution to model latent user-specific parameters.
  3. Bootstrap Difference-in-Differences (DiD): This component ensures that observed changes are not merely site-wide trends or random fluctuations.
The framework utilizes a sliding window and virtual badges to control for random fluctuations in user behavior over time.

As illustrated in the methodology diagram, the "true" badge introduction is compared against "virtual badges" placed at random points in time. If the behavioral shift at the actual introduction significantly exceeds these random fluctuations, the effect is deemed causal.

Experimental Setup

The researchers analyzed a massive dataset from Stack Overflow spanning 2008 to 2014. They focused on three specific badges with varying levels of intrinsic utility:

  • Tag Editor: Awarded for editing a "tag wiki" (Low intrinsic utility; high community effort).
  • Investor: Awarded for offering a bounty on someone else's question (Medium utility).
  • Promoter: Awarded for offering a bounty on one's own question (High intrinsic utility; direct personal benefit).
Visualization of the natural experiment for the Tag Editor badge, showing user action times relative to the badge introduction.

The figure above shows the distribution of user actions relative to the introduction of the Tag Editor badge. This temporal boundary is critical for isolating the badge's effect from general site growth.

Key Results and Insights

The study's most significant finding is the existence of a "utility threshold." Badges appear to be effective only when the intrinsic utility of the rewarded action is low.

  • Causal Success: The Tag Editor ($p=0.004$) and Investor ($p=0.017$) badges showed significant causal effects. In contrast, the Promoter badge ($p=0.309$) did not, as users were already motivated to perform this high-utility action.
  • Impact Magnitude: For successful badges, their introduction led to a 4x increase in the average intensity of the rewarded action and a 75% reduction in the time it took for a user to perform the action for the first time.
  • Steering Effect: The Tag Editor badge was particularly effective for "unpopular" tags, suggesting that badges are most useful for steering users toward neglected but necessary community tasks.

Conclusion

This research provides a robust framework for quantifying the impact of digital rewards. The findings suggest that badges are not a universal solution but are highly effective at motivating users to perform low-utility, community-oriented tasks. While the current model is limited to first-time badges, it sets the stage for future work on more complex reward structures, such as threshold-based badges, and provides a principled approach for the design of gamified systems.

Summary updated

Abstract

A wide variety of online platforms use digital badges to encourage users to take certain types of desirable actions. However, despite their growing popularity, their causal effect on users' behavior is not well understood. This is partly due to the lack of counterfactual data and the myriad of complex factors that influence users' behavior over time. As a consequence, their design and deployment lacks general principles. In this paper, we focus on first-time badges, which are awarded after a user takes a particular type of action for the first time, and study their causal effect by harnessing the delayed introduction of several badges in a popular Q&A website. In doing so, we introduce a novel causal inference framework for badges whose main technical innovations are a robust survival-based hypothesis testing procedure, which controls for the utility heterogeneity across users, and a bootstrap difference-in-differences method, which controls for the random fluctuations in users' behavior over time. We find that first-time badges steer users' behavior if the utility a user obtains from taking the corresponding action is sufficiently low, otherwise, the badge does not have a significant effect. Moreover, for badges that successfully steered user behavior, we perform a counterfactual analysis and show that they significantly improved the functioning of the site at a community level.
BibTeX
@misc{kusmierczyk2017harnessing,
  author = {Tomasz Kusmierczyk and Manuel Gomez-Rodriguez},
  title = {{Harnessing Natural Experiments to Quantify the Causal Effect of Badges}},
  year = {2017},
  eprint = {1707.08160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.08160},
}
Figure