On Validation and Predictability of Digital Badges' Influence on Individual Users

Tomasz Kusmierczyk, Kjetil NørvågAAAI 2018 · OpenReview

Abstract

Badges are a common, and sometimes the only, method of incentivizing users to perform certain actions on online sites. However, due to many competing factors influencing user temporal dynamics, it is difficult to determine whether the badge had (or will have) the intended effect or not. In this paper, we introduce two complementary approaches for determining badge influence on users. In the first one, we cluster users’ temporal traces (represented with Poisson processes) and apply covariates (user features) to regularize results. In the second approach, we first classify users’ temporal traces with a novel statistical framework, and then we refine the classification results with a semi-supervised clustering of covariates. Outcomes obtained from an evaluation on synthetic datasets and experiments on two badges from a popular Q&A platform confirm that it is possible to validate, characterize and to some extent predict users affected by the badge.

Introduction

Digital badges are ubiquitous in gamified systems like Stack Overflow, Khan Academy, and Foursquare. While intended to motivate users, it is often unclear if they act as genuine incentives or merely as "participation trophies" for users who would have been active regardless. The core challenge lies in the lack of "ground truth": observing an increase in activity as a user nears a badge threshold could be due to the badge (the "goal-gradient hypothesis") or simply natural user evolution and external factors. This research addresses the user badge influence problem at the individual level, focusing on two tasks: Validation (did a badge recipient change their behavior to earn it?) and Prediction (can we forecast if a new user will be motivated by a badge based on their profile?).

Methodology

The study models user activity as non-homogeneous Poisson processes, tracking how the rate of actions changes over time. Two primary approaches are proposed:

  1. Poisson Processes Clustering: This model-based approach clusters users into "attracted" and "not attracted" groups, using user features (covariates) to regularize results and distinguish badge-related spikes from random noise.
  2. 2-Phase Bootstrap: This robust framework first uses Null Hypothesis Significance Testing (NHST) with a "virtual badges" bootstrapping technique. By simulating "fake" badge awarding times, it creates a counterfactual distribution to assess the statistical significance of behavior changes. The second phase refines these results using a semi-supervised EM algorithm that incorporates user covariates such as age, location, and site statistics.
User activity trace and associated covariates used in the modeling process.

The models were evaluated using synthetic data and real-world data from Stack Overflow (2008–2014), specifically focusing on the Research Assistant (50 tag wiki edits) and Copy Editor (500 post edits) badges. User covariates included age, location (via word2vec embeddings), and site statistics like views, comments, and upvotes.

Key Results and Insights

The evaluation revealed that a significant portion of users are "accidental achievers." For the Copy Editor badge, only 47% of earners were found to be genuinely "attracted" by the reward. Demographic analysis suggested that younger users and users from specific regions (e.g., Eastern Europe and India) tend to be more goal-oriented and driven by badge rewards compared to users in the USA, who often receive badges "by chance" due to higher baseline activity.

Validation and prediction performance (AUC) across varying global trend strengths.

The 2-phase bootstrap method consistently outperformed simple Poisson clustering, particularly in the presence of external trends. While validation of past behavior was successful (estimating 58% attraction for Research Assistant), predicting future attraction for new users proved more challenging, with AUC scores ranging from 0.53 to 0.63. This indicates that while user features provide some signal, temporal activity remains the most potent indicator of badge influence.

Discussion

A primary limitation of these models is the assumption that badge attraction manifests as a shift in the mean rate of actions, potentially overlooking subtler psychological influences. Future work could explore more advanced features, such as early-stage temporal patterns, to improve predictive accuracy. Additionally, while this study focused on "threshold badges," different modeling techniques may be required for other badge types, such as those with limited availability.

Summary updated

Abstract

Badges are a common, and sometimes the only, method of incentivizing users to perform certain actions on on- line sites. However, due to many competing factors influencing user temporal dynamics, it is difficult to determine whether the badge had (or will have) the intended effect or not. In this paper, we introduce two complementary approaches for determining badge influence on users. In the first one, we cluster users’ temporal traces (represented with Poisson processes) and apply covariates (user features) to regularize results. In the second approach, we first classify users’ temporal traces with a novel statistical framework, and then we refine the classification results with a semi-supervised clustering of covariates. Outcomes obtained from an evaluation on synthetic datasets and experiments on two badges from a pop- ular Q&A platform confirm that it is possible to validate, characterize and to some extent predict users affected by the badge.
BibTeX
@inproceedings{kusmierczyk2018validation,
  author = {Tomasz Kusmierczyk and Kjetil Nørvåg},
  title = {{On Validation and Predictability of Digital Badges' Influence on Individual Users}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=HJZdHxbu-H},
}
Figure