Augmenting Survey Data with Digital Trace Data: Is There a Threat to Panel Retention?

Mark Trappmann, Georg‐Christoph Haas, Sonja Malich, Florian Keusch, Sebastian Bähr, Frauke Kreuter, Štefan SchwarzJournal of Survey Statistics and Methodology 2022 · DOI

TL;DR

Asking long-term survey participants to install a smartphone tracking app causes a small, immediate dip in future participation (about 3.3 percentage points). However, this "chilling effect" is temporary; by the second and third years after the request, there is no significant difference in retention between those asked and those left alone.

What Problem Does This Paper Address?

Social scientists are increasingly looking to "digital trace data"—information from GPS sensors, app usage logs, and communication metadata—to supplement traditional surveys. While surveys rely on fallible human memory, digital traces offer high-precision, objective behavioral data.

The catch? Collecting this data requires intrusive requests, such as asking participants to install a research app that monitors their daily lives for months. Survey "panels" (groups of people interviewed repeatedly over years) are incredibly expensive to maintain. Researchers fear that these high-burden requests might "poison the well," causing participants to quit the panel entirely (attrition), thereby destroying the longitudinal value of the study. This paper quantifies exactly how much of a threat these requests pose to panel retention.

Methods

The authors embedded a randomized experiment within Wave 11 of the German PASS panel survey.

  • The Setup: They identified 5,721 eligible participants who owned Android smartphones.
  • The Intervention: 75% of the group (the treatment group) were invited to participate in the "IAB-SMART" study, which required installing an app for six months to collect sensor and log data. The remaining 25% (the control group) received no such invitation.
  • The Analysis: The researchers tracked both groups over the next three years (Waves 12, 13, and 14 of the PASS survey). They used a linear probability model with cluster-robust standard errors to determine if the invitation itself caused people to drop out of the main survey, regardless of whether they actually installed the app.

Datasets

  • PASS (Panel Study Labour Market and Social Security): A massive, mixed-mode longitudinal survey in Germany active since 2007, focusing on the labor market and welfare.
  • IAB-SMART: A custom Android research app designed to collect five categories of data: network quality/location, interaction history, social network characteristics, activity data (accelerometer), and smartphone usage.

Key Insights and Ideas

  • The "Invitation Effect" vs. the "Participation Effect": The study focuses on the effect of the invitation. Interestingly, the people who actually installed the app were the most likely to stay in the panel long-term. The attrition was driven by people who were invited but declined. This suggests that the request acts as a "stress test" for participant commitment.
  • Accelerated Attrition: The authors hypothesize that the invitation might not be creating new dropouts so much as speeding up the departure of people who were already on the fence. This explains why the negative effect disappears after the first year—the "unfaithful" participants simply left a wave earlier than they otherwise would have.
  • Privacy vs. Burden: Previous research cited in the paper suggests that for digital trace data, participants are more worried about privacy and security than the actual effort of installing the app. This distinguishes app-based requests from other "burdensome" tasks like longer questionnaires.

Key Results

  • Immediate Impact: In the year immediately following the invitation (Wave 12), retention in the treatment group was 3.3 percentage points lower than the control group (statistically significant at $p < .01$).
  • Long-term Recovery: By Wave 13, the gap narrowed to 2.3 points, and by Wave 14, it was only 1.5 points. Neither of these later differences were statistically significant.
  • Participation Rates: Only 15.9% of those invited actually installed the app, despite relatively high incentives (up to 80 Euros).
  • The "Loyalty" Gap: There was a massive 15.7 percentage point difference in retention between those who participated in the app study and those who were never invited. This confirms a strong selection bias: the people willing to give you their digital trace data are your most dedicated "super-respondents."

Limitations and Future Work

  • Subgroup Analysis: The study lacked the statistical power to determine if the invitation effect is worse for specific demographics (e.g., older participants or those with higher privacy concerns).
  • Incentive Structures: The study used high incentives for the app. It remains unclear if lower incentives would lead to higher attrition from the main panel due to a perceived lack of fairness.
  • Panel Maturity: The PASS panel was over a decade old when this experiment started. Participants were already "vetted" and loyal. A newer panel might see much higher attrition rates when hit with a high-intensity data request.
  • Optimization: Future research is needed to see if the timing of the invitation (e.g., at the end of a survey vs. a separate mailing) can mitigate the initial 3.3% drop.

Summary updated

Abstract

Abstract Linking digital trace data to existing panel survey data may increase the overall analysis potential of the data. However, producing linked products often requires additional engagement from survey participants through consent or participation in additional tasks. Panel operators may worry that such additional requests may backfire and lead to lower panel retention, reducing the analysis potential of the data. To examine these concerns, we conducted an experiment in the German PASS panel survey after wave 11. Three quarters of panelists (n = 4,293) were invited to install a research app and to provide sensor data over a period of 6 months, while one quarter (n = 1,428) did not receive an invitation. We find that the request to install a smartphone app and share data significantly decreases panel retention in the wave immediately following the invitation by 3.3 percentage points. However, this effect wears off and is no longer significant in the second and third waves after the invitation. We conclude that researchers who run panel surveys have to take moderate negative effects on retention into account but that the potential gain likely outweighs these moderate losses.
BibTeX
@article{trappmann2022augmenting,
  author = {Mark Trappmann and Georg‐Christoph Haas and Sonja Malich and Florian Keusch and Sebastian Bähr and Frauke Kreuter and Štefan Schwarz},
  title = {{Augmenting Survey Data with Digital Trace Data: Is There a Threat to Panel Retention?}},
  journal = {Journal of Survey Statistics and Methodology},
  year = {2022},
  doi = {10.1093/jssam/smac023},
}
Figure