Mobile health apps send people prompts, such as "take a short walk now," and the effect of each prompt fades as people get used to them. Every major system in this space treats that fading as a problem of volume. They cap the number of messages per day, track a decaying count of recent messages, and filter out repeats. Learning theory points to a second cause that nobody has measured. Behavior that is reinforced on an unpredictable schedule fades more slowly than behavior reinforced on a regular one; psychologists call this a variable-ratio schedule when rewards arrive after an unpredictable number of responses. If the same holds for health prompts, an app could stay effective by making its timing less predictable, without sending more messages.
We propose to test this using data that already exists. Micro-randomized trials, where a coin flip at each decision point decides whether to send a prompt, leave behind logs that can be analyzed like a retrospective cohort: a historical record of people followed forward from an exposure. These logs have one unusual property. Part of the variation in how irregular each person's prompt history was came from the known coin flips. Studies that compare self-selected users cannot isolate that kind of variation. We use it to ask whether timing predictability changes how much prompts lose their effect, after accounting for how many prompts a person received.
Core Claim
Holding prompt dose fixed, people whose recent prompt history was more irregular will show a slower decline in their step-count response to prompts than people with regular histories. They will also show more persistent activity during randomly occurring stretches with no prompts. Because the randomization probabilities are logged, this can be estimated from existing trial data without the self-selection problem that would sink the obvious design. The obvious design is a registry comparison of people who opted into lottery-style incentives against people who did not.
The claim is deliberately narrower than "variable-ratio reinforcement improves health." Prompts are cues to act, not rewards for acting. What we test is the mechanism the reinforcement literature attributes to unpredictable schedules: people cannot tell a pause from the end of the program, so their behavior persists. We test it in the one real-world setting where the schedule's irregularity was partly randomized. Showing that this mechanism holds for prompt timing would justify a prospective trial of true variable-ratio rewards. Showing that it does not would remove a frequently assumed rationale.
Why This Gap, and Not the Obvious One
Two strands of work motivate the question.
The first suggests that unpredictability matters. In Adventures of AI Directors Early in the Development of Nightingale, the authors tested a bandit-based personalizer (an algorithm that learns which quests each player prefers) against uniform random quest offers in a video game. Random offers produced more total playtime: 262.7 minutes versus 211.3. The authors attribute this to intermittent high-value encounters. How do we reinforce climate action? synthesizes evidence that lottery-style incentives resist the indifference that small fixed incentives produce. The extinction-resistance principle itself traces to older clinical conditioning work such as Aversion therapy in management of 43 homosexuals.. None of these settings involved health behavior, and none separated predictability from dose.
The second strand designs against fatigue purely through volume. Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity models habituation with a "dosage" variable, an exponentially decaying count of recent prompts, and clips send probabilities to between 0.1 and 0.8. Evaluation of the HeartSteps Online Sampling Algorithm enforces a budget of 1.5 prompts per day, spread uniformly. NudgeRank: Digital Algorithmic Nudging for Personalized Health applies recency and fatigue filters to roughly a million users. In all of these, predictability is either absent from the model or deliberately suppressed: uniform spreading of prompts makes timing more regular.
The research file contains no retrospective study that treats reinforcement schedule as the exposure. The natural version of that study would compare health-plan members who used lottery features against those who used fixed rewards, with propensity scores. A propensity score is the estimated probability of being exposed given baseline traits, used to make groups comparable. This method is the workhorse of Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach and the registry analysis in Practice patterns and outcomes for patients with node-negative hormone receptor-positive breast cancer and intermediate 21-gene Recurrence Scores. It is the obvious design, and we think it is the wrong first study. Choosing a gamified feature is driven by motivation and novelty-seeking, which no medical record captures. Those same traits plausibly predict persistence directly. Balancing on observed covariates cannot remove that bias. Our design keeps the cohort logic but sources the exposure variation from randomization.
Research Program
The data are the trial logs indexed in the file. HeartSteps V2V3 provides the main analysis: 82 adults with stage 1 hypertension, 9,423 participant-days, about 1.3 million decision points, and 10,137 delivered prompts, with logged probabilities. HeartSteps V1 (44 adults, 42 days) serves as a replication set, as analyzed in Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health. The Oralytics-parameterized simulator from Replicable Bandits for Digital Health Interventions is used for method validation, because there the true effect can be planted.
At each decision point we compute two summaries of the person's recent prompt history over a fixed window, such as seven days. The first is dose: the HeartSteps exponentially decaying count. The second is irregularity: the coefficient of variation of the gaps between prompts, and as a robustness check, the entropy of prompt timing across time-of-day bins. These two summaries are correlated, and irregularity is also confounded by availability. People with chaotic days are unavailable more often and have more irregular histories for that reason alone.
The key step is to recenter the exposure. For each person and window, we simulate the trial's randomization many times. Each simulation keeps the realized availability, the realized states, and the logged send probabilities, and draws new coin flips. This gives the irregularity a person would have had on average, given everything that was not randomized. Realized irregularity minus this expected irregularity is, by construction, driven only by coin flips, and so it is independent of the person's motivation and daily chaos. We do the same for dose.
We then estimate the effect of each recentered quantity on the prompt's proximal effect, the change in 30-minute step count after a prompt. We use the action-centered weighted estimator already standard for these trials, with the recentered irregularity and dose entered as moderators. This is a valid comparison only because the logged probabilities are known and bounded away from zero and one. The replicable-bandits paper shows why that condition matters: with greedy algorithms, estimators fail to settle on a single answer.
The extinction analogue uses randomly arising gaps. Runs of consecutive decision points where available people received no prompt arise from the coin flips. For each run we compare daily steps during the gap with steps before it, and test whether prior irregularity predicts a smaller drop.
The final product is an off-policy estimate of a policy that has never been deployed. It matches the trial's 1.5-prompts-per-day budget but deliberately varies inter-prompt gaps. We compare its estimated long-term step value to the uniform-spreading policy at the same dose.
Baselines
The naive reference point is the model that HeartSteps already assumes: habituation depends on dose alone. We fit the dose-moderated proximal-effect model from Personalized HeartSteps on the same V2V3 data, with the same estimator and the same window choices. The proposal beats it only if adding recentered irregularity improves held-out-participant prediction of proximal effects. The irregularity coefficient must also have a confidence interval excluding zero after we account for having tested several windows.
The confounded baseline regresses the proximal effect on raw, un-recentered irregularity. It shows what a conventional observational analysis of these logs would conclude. If raw and recentered estimates diverge sharply, that divergence is itself evidence that the obvious cohort design would mislead.
The strongest existing method for asking whether a policy is better over the long run is the projected-Bellman off-policy estimator of Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health. On HeartSteps V1 it reports long-term average rewards of 2.962 for "do nothing," 3.127 for "always treat," and 3.155 for location-based treatment. The authors translate the last into roughly 55 more steps per day than doing nothing, with no significant difference from always treating. We run that estimator unchanged, with its default state, and then with irregularity added to the state, to evaluate the dose-matched high-irregularity policy against the uniform policy. Beating the baseline means the high-irregularity policy's estimated value exceeds the uniform policy's with a confidence interval excluding zero. Its advantage must also disappear when irregularity is removed from the state, which shows that the gain depends on modeling predictability. That removal is the ablation for our key ingredient.
A second ablation drops the recentering step but keeps everything else. This isolates the contribution of using randomization-only variation.
In the Oralytics simulator, the bar is concrete. With a planted predictability effect, the estimator must recover it with close to 95% interval coverage, the coverage that replicable algorithms achieve in that paper. With no planted effect, it must find none, even when availability is engineered to correlate with irregularity.
Why not just…?
Why not just run a randomized trial of variable versus fixed reward schedules? That is the right confirmatory study, and this proposal is a real competitor to doing it immediately. The case for analyzing logs first is cost and design. Existing trials have small samples of 8 to 82 people. A schedule trial powered for a habituation-slope difference needs an effect-size estimate that nobody has. Our analysis supplies that estimate, or a reason not to run the trial at all. If the recentered effect is null with a tight interval, a prospective trial is poorly motivated.
Why not just control for total dose and call it done? If fatigue is purely about volume, the dose-only model will match ours on held-out prediction, and the irregularity coefficient will be near zero once dose is recentered too. We report exactly that comparison. The simpler explanation wins if the data say so. The one sign of more than dose already in the file is the Nightingale result, which came from a different domain and did not hold dose fixed, so it cannot settle the question.
Why not just do the registry study of lottery-feature users with propensity matching? The prostate and breast cancer cohorts show that propensity methods can balance measured severity. The confounder here is unmeasured: intrinsic motivation and novelty-seeking, which plausibly drive both opting into lotteries and sustaining activity. Our confounded-versus-recentered comparison directly measures how much bias arises from not using randomization in this exact setting. That number can be quoted to anyone proposing the registry design.
Why not just apply standard off-policy evaluation? We do, as the main baseline. Its default state does not include history irregularity, and it assumes time-homogeneity, an assumption its authors flag as strained by engagement decay. Adding irregularity to the state tests whether that limitation hides a predictability effect.
Decisive Checks
Three results would settle the question. The first is a placebo test: irregularity in the window after a decision point should not predict that point's proximal effect, and any association there signals leakage in the recentering. The second is replication: the sign of the effect estimated on V2V3 must hold on HeartSteps V1, in the different, healthy, short-trial population. The third is the gap analysis: if prior irregularity predicts smaller step declines during randomly arising no-prompt stretches, that is the field analogue of resistance to extinction.
The main threat is statistical power. Only 82 people contributed, and the variation that comes from randomization alone is a fraction of total irregularity. We will report the minimum detectable effect in advance, using the simulator calibrated to V2V3 variance. We will interpret a null result relative to that threshold, not as proof of no effect.
References
- Adventures of AI Directors Early in the Development of Nightingale
- How do we reinforce climate action?
- Aversion therapy in management of 43 homosexuals.
- Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity
- Evaluation of the HeartSteps Online Sampling Algorithm
- NudgeRank: Digital Algorithmic Nudging for Personalized Health
- Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach
- Practice patterns and outcomes for patients with node-negative hormone receptor-positive breast cancer and intermediate 21-gene Recurrence Scores
- Off-Policy Estimation of Long-Term Average Outcomes With Applications to Mobile Health
- Replicable Bandits for Digital Health Interventions