TL;DR
Asking long-term survey participants to install a smartphone tracking app causes a small, immediate dip in future participation (about 3.3 percentage points). However, this "chilling effect" is temporary; by the second and third years after the request, there is no significant difference in retention between those asked and those left alone.
What Problem Does This Paper Address?
Social scientists are increasingly looking to "digital trace data"—information from GPS sensors, app usage logs, and communication metadata—to supplement traditional surveys. While surveys rely on fallible human memory, digital traces offer high-precision, objective behavioral data.
The catch? Collecting this data requires intrusive requests, such as asking participants to install a research app that monitors their daily lives for months. Survey "panels" (groups of people interviewed repeatedly over years) are incredibly expensive to maintain. Researchers fear that these high-burden requests might "poison the well," causing participants to quit the panel entirely (attrition), thereby destroying the longitudinal value of the study. This paper quantifies exactly how much of a threat these requests pose to panel retention.
Methods
The authors embedded a randomized experiment within Wave 11 of the German PASS panel survey.
- The Setup: They identified 5,721 eligible participants who owned Android smartphones.
- The Intervention: 75% of the group (the treatment group) were invited to participate in the "IAB-SMART" study, which required installing an app for six months to collect sensor and log data. The remaining 25% (the control group) received no such invitation.
- The Analysis: The researchers tracked both groups over the next three years (Waves 12, 13, and 14 of the PASS survey). They used a linear probability model with cluster-robust standard errors to determine if the invitation itself caused people to drop out of the main survey, regardless of whether they actually installed the app.
Datasets
- PASS (Panel Study Labour Market and Social Security): A massive, mixed-mode longitudinal survey in Germany active since 2007, focusing on the labor market and welfare.
- IAB-SMART: A custom Android research app designed to collect five categories of data: network quality/location, interaction history, social network characteristics, activity data (accelerometer), and smartphone usage.
Key Insights and Ideas
- The "Invitation Effect" vs. the "Participation Effect": The study focuses on the effect of the invitation. Interestingly, the people who actually installed the app were the most likely to stay in the panel long-term. The attrition was driven by people who were invited but declined. This suggests that the request acts as a "stress test" for participant commitment.
- Accelerated Attrition: The authors hypothesize that the invitation might not be creating new dropouts so much as speeding up the departure of people who were already on the fence. This explains why the negative effect disappears after the first year—the "unfaithful" participants simply left a wave earlier than they otherwise would have.
- Privacy vs. Burden: Previous research cited in the paper suggests that for digital trace data, participants are more worried about privacy and security than the actual effort of installing the app. This distinguishes app-based requests from other "burdensome" tasks like longer questionnaires.
Key Results
- Immediate Impact: In the year immediately following the invitation (Wave 12), retention in the treatment group was 3.3 percentage points lower than the control group (statistically significant at $p < .01$).
- Long-term Recovery: By Wave 13, the gap narrowed to 2.3 points, and by Wave 14, it was only 1.5 points. Neither of these later differences were statistically significant.
- Participation Rates: Only 15.9% of those invited actually installed the app, despite relatively high incentives (up to 80 Euros).
- The "Loyalty" Gap: There was a massive 15.7 percentage point difference in retention between those who participated in the app study and those who were never invited. This confirms a strong selection bias: the people willing to give you their digital trace data are your most dedicated "super-respondents."
Limitations and Future Work
- Subgroup Analysis: The study lacked the statistical power to determine if the invitation effect is worse for specific demographics (e.g., older participants or those with higher privacy concerns).
- Incentive Structures: The study used high incentives for the app. It remains unclear if lower incentives would lead to higher attrition from the main panel due to a perceived lack of fairness.
- Panel Maturity: The PASS panel was over a decade old when this experiment started. Participants were already "vetted" and loyal. A newer panel might see much higher attrition rates when hit with a high-intensity data request.
- Optimization: Future research is needed to see if the timing of the invitation (e.g., at the end of a survey vs. a separate mailing) can mitigate the initial 3.3% drop.
Summary updated