Home-based digital therapies, such as cognitive training games for children with attention deficits or puzzle regimens for older adults, lose most of their users. Reported dropout runs from 30% to 90%. A natural remedy is dynamic difficulty adjustment: a control loop that keeps each task hard enough to be useful but not so hard that the person quits. Learning a good difficulty policy requires pooling experience across many patients, and privacy rules push that pooling toward federated learning, where each device trains on its own data and sends only model updates to a server. This note argues that combining the two creates a specific failure. In a federated system, the users who contribute updates are the ones who stayed. Whether they stayed depends partly on the difficulty the shared model gave them. The aggregated model therefore gradually tunes itself to the people who tolerate it. It then serves that survivor-tuned difficulty to new, more fragile users, who leave faster, which strengthens the bias further.
The core claim is that client dropout in federated difficulty adjustment is caused by the policy itself and carries information about it. This breaks the assumption behind existing evidence that federated health models tolerate dropout, and a correction borrowed from survival analysis can repair it without moving raw data off the device. Concretely, each device reweights its update by the inverse of its estimated probability of having reached its current stage of the program. It applies that weight locally, before secure aggregation. We will show when the bias appears, how large it is, and whether this reweighting closes the gap to a centralized model that sees everyone's data.
Why this gap is real
Three bodies of evidence motivate the proposal.
First, federated learning in health has been validated under a benign model of dropout. Privacy-first health research with federated learning replicated eight clinical studies with federated models. It found that training reached 99% of maximum performance with only 2% of devices present per round. That dropout was treated as random network loss. In a therapy app, dropout is the outcome being studied, and it is correlated with the model's own outputs.
Second, pooling matters in noisy behavioral data. A Deployed Online Reinforcement Learning Algorithm in an Oral Health Clinical Trial found that a fully pooled reward model outperformed individual models in counterfactual simulation. A centralized pool keeps early-session data from users who later quit. Federated rounds only ever see whoever is still enrolled.
Third, who survives is not neutral. Association Between Task Difficulty Regulation in Game-Based Digital Therapeutics and Attention-Deficit/Hyperactivity Disorder Symptoms in Children analyzed only children who completed the program. It found that lower, stabler, and downward-sloping difficulty tracked symptom improvement. That paper constrains our design in two ways. The controller must not reward escalation for its own sake. Its completers-only analysis is also an instance of the very survivorship problem we study.
Two further lines of work supply our components. A Temporal Data-Driven Player Model for Dynamic Difficulty Adjustment showed that a two-component factorization of user × skill × time explains over 90% of performance variance. That makes it possible to forecast skill proactively, rather than reacting after a failure. We reuse this as the shared skill model and move its fitting into a federated setting. Early Attrition Prediction for Web-Based Interpretation Bias Modification to Reduce Anxious Thinking showed that passive first-session telemetry, such as task latencies and page-load times, predicts return for session two. XGBoost reached an F1-macro of 0.917 on the Calm Thinking trial, falling to 0.832 and 0.770 on the other two trials. We reuse this kind of signal for two jobs: estimating each user's dropout hazard, and computing the survival weights.
Core Claim
Take a federated difficulty controller whose clients leave the program at a rate that depends on the difficulty mismatch they experience. Plain federated averaging then converges to a policy that is systematically too hard for users with slow skill growth. Their retention falls below what a centralized model on the same population achieves. This gap should vanish when dropout is made random, which is the setting studied by Sadilek et al. It should shrink under stage-stratified inverse-survival weighting.
Research Program
The system has three shared components, all learned federatedly.
The first is a skill-trajectory model in the style of Zook and Riedl. The global factors live on the server, while each user's loadings stay on their device.
The second is a dropout-hazard model built on early telemetry and recent challenge mismatch.
The third is the controller. At each session it picks the difficulty whose predicted success rate lies inside a target window, such as the performance-window designs of SIMPLE or the 70% target in NEUROCRYPT. It breaks ties toward lower predicted hazard and toward smaller difficulty changes, following Kim et al. The controller is deliberately simple. The research contribution is how its shared parts are aggregated.
Updates are grouped by program stage $k$, meaning session index or week. For stage $k$, the server forms
$$\bar g_k = \frac{\sum_{i \in P_k} g_{i,k}/\hat S_k(x_i)}{\sum_{i \in P_k} 1/\hat S_k(x_i)},$$
where the symbols are defined as follows. $P_k$ is the set of clients sending updates from stage $k$. $g_{i,k}$ is client $i$'s update at that stage, capped so each client contributes one unit of weight per stage no matter how long it stays. $\hat S_k(x_i)$ is the probability, under the current hazard model, that a user with first-session features $x_i$ reaches stage $k$. It is clipped away from zero.
Each device computes its own weight and multiplies before secure aggregation. The server therefore learns only the weighted sum and the total weight, never individual weights. Differential-privacy noise is added as in Sadilek et al.
This is the standard inverse-probability correction for monotone dropout. It is valid only if dropout depends on things the device observes, the "missing at random" assumption. The hazard model's calibration on held-out users is therefore a first-class output. We also add a sensitivity analysis in which dropout depends on an unobserved factor, to see how quickly the correction degrades.
The first deliverable is a simulator with real enrollment inflow and policy-dependent exits. Skill trajectories come from a two-component factor model with a mix of fast and slow learners. The per-session dropout hazard rises with the gap between the offered difficulty and current skill, in both directions, to capture frustration and boredom. Hazard parameters are set so that overall attrition spans the 30% to 90% range reported by Baee et al. and by New Approach to Equitable Intervention Planning to Improve Engagement and Outcomes in a Digital Health Program.
The second deliverable is a real-data test on the MindTrails trials. These trials have genuine attrition labels and telemetry but no difficulty adjustment. We will replay users as federated clients who take part in rounds only while enrolled. This measures how tenure-weighted aggregation distorts the attrition predictor, especially on early leavers, compared with centralized XGBoost. The three trials also act as silos for a cross-silo variant, since Baee et al. found that models transfer poorly between them.
Baselines
The naive reference point is a local performance staircase: raise difficulty after sustained success, lower it after failure, aiming for a fixed success rate, with no learning and no data sharing. It is what many deployed systems effectively do, and it carries no privacy cost. It runs on the same simulated users and the same MindTrails replay sequences, with its step size and target tuned on the same validation budget as every learned method. No reported result was found on Lacuna for a staircase on these benchmarks.
The upper reference is centralized full pooling in the Oralytics sense. The same controller and models are trained on all users' complete histories, including users who left. This is the oracle that our method tries to approach, not beat. The only related number in the file comes from a different task: 69.724 average Oral Self-Care Behavior score for full pooling versus 69.375 for no pooling, in counterfactual simulation, in Trella et al. On MindTrails, the centralized comparison point is Baee et al.'s F1-macro of 0.917 (Calm Thinking), 0.832 (Managing Anxiety), and 0.770 (Future Thinking).
The strongest federated competitor is plain federated averaging with sample-count weighting. We also run a personalized variant in the style of FedHealth, which shares lower layers and adapts a local head, and which reported 99.4% activity-recognition accuracy, 5.3 points above its non-personalized baseline. Personalization is the most likely thing a reviewer expects to absorb the bias, because each user gets their own adjustment. A purely local no-pooling model completes the set. All methods share round counts, participation rates, noise levels, and hyperparameter search budgets.
Three ablations isolate the key ingredient. The first keeps per-client capping but drops the inverse-survival weights. The second keeps the weights but drops stage stratification. The third keeps both but removes the hazard term from the controller, which separates "better aggregation" from "a more cautious controller."
The success criterion has two parts. Our method must recover at least half the gap between federated averaging and the centralized oracle in week-four retention among slow learners. It must do so without lowering retention among fast learners and without raising difficulty slope or variability. It must also beat the staircase on the same retention metric.
Why not just…?
Why not just centralize de-identified logs, as Oralytics and PEARL did? Centralization is the oracle in our experiments, so it will win on accuracy by construction. The research file argues that millisecond-level interaction telemetry works as a behavioral fingerprint and resists simple de-identification (Privacy-first health research with federated learning). It also notes that health data often sit in legal or commercial silos (FedHealth). Our question is conditional: if a deployment must be federated, what does it cost, and can the cost be recovered? We report the centralized number throughout, so readers can judge the trade-off directly.
Why not just run a local staircase and skip learning entirely? This is a real threat. If users' skill growth is regular enough, a well-tuned staircase may match any learned controller, and the survivor-bias question becomes moot. Two things give us reason to doubt it. Staircases react only after failure, whereas proactive forecasting in Zook and Riedl anticipated performance from a few sessions. In PEARL, fixed rule-based personalization of nudges did not beat random nudges (PEARL), though that result concerns messages, not difficulty. The simulator settles the matter empirically. If the staircase matches the centralized oracle's slow-learner retention, we will report that pooling is unnecessary for this task and stop there.
Why not just train a separate model on each device? Oralytics found no-pooling worse than full pooling. More importantly, the highest-risk window is between the first and second sessions, when a local model has almost no data (Baee et al.). No-pooling is also immune to survivor bias. Comparing it with plain federated averaging tells us whether biased pooling can end up worse than no pooling at all, which would be a striking negative result in its own right.
Why not just trust the 2% participation result, or keep dropped clients' last updates on the server? The 2% result assumes random absence. Our decisive control reproduces that random setting and predicts no gap there. Keeping stale updates is a cheap and serious alternative. It preserves some influence from users who have left, but it does not upweight at-risk users who are still present and underrepresented in later stages. We include it as a baseline. If it closes the gap as well as inverse-survival weighting does, we will recommend it instead, because it is simpler.
Proposed Experiments
The decisive check comes first. In the simulator, we compare federated averaging against the centralized oracle under two conditions: random dropout at matched overall rates, and policy-induced dropout. The premise is falsified if the retention gap for slow learners is negligible in both conditions.
If the gap exists, we run all baselines and ablations. We sweep attrition severity from 30% to 90%, the share of slow learners, participation per round down to 2%, and privacy-noise levels. The metrics are per-subgroup survival curves, time spent inside the target success window, difficulty mean, variability, and slope as defined by Kim et al., and the retention disparity between learner subgroups. The disparity measure is reported in the same spirit as the equity measures in Killian et al.
On MindTrails, we measure how tenure-weighted federated training changes F1-macro on early leavers compared with Baee et al.'s centralized results. We then test whether inverse-survival weighting restores it, both within each trial and across the three trials treated as silos.
A final robustness test makes the true dropout depend on an unobserved factor. This measures how the correction degrades when its key assumption fails.
A small prospective home pilot, randomizing the aggregation rule, is proposed only if the simulator shows a gap of practical size. The Urwyler pilot's n=12 and the Kim trial's n=35 make clear how underpowered such pilots usually are. We would therefore power the pilot on slow-learner retention alone.
References
- Privacy-first health research with federated learning
- A Deployed Online Reinforcement Learning Algorithm in an Oral Health Clinical Trial
- Association Between Task Difficulty Regulation in Game-Based Digital Therapeutics and Attention-Deficit/Hyperactivity Disorder Symptoms in Children: Secondary Analysis of a Randomized Controlled Trial
- A Temporal Data-Driven Player Model for Dynamic Difficulty Adjustment
- Early Attrition Prediction for Web-Based Interpretation Bias Modification to Reduce Anxious Thinking: A Machine Learning Study
- Shanghai cognitive intervention of mild cognitive impairment for delaying progress with longitudinal evaluation-a prospective, randomized controlled study (SIMPLE): rationale, design, and methodology
- NEUROCRYPT: Coercion-Resistant Implicit Memory Authentication (Student Abstract)
- New Approach to Equitable Intervention Planning to Improve Engagement and Outcomes in a Digital Health Program: Simulation Study
- FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
- A Personalized Exercise Assistant using Reinforcement Learning (PEARL): Results from a four-arm Randomized-controlled Trial