Separating Device Noise from Disengagement to Transfer Dropout Predictors Across Digital Interventions

Self-guided mental health apps lose most of their users early. Between 30% and 90% of enrolled participants stop before finishing the program. A model that flags likely dropouts after the first session would let a program spend scarce human coaching where it matters. Such models already exist, but they work only inside the trial they were trained on. Early Attrition Prediction for Web-Based Interpretation Bias Modification predicted from Session 1 behavior (page dwell time, task latency, page-load time, device type) whether a user would return for Session 2. Its predictor reached an F1-macro of 0.917 on one trial, yet it "did not transfer perfectly" to two sibling trials that ran on the same web platform. A newly launched intervention therefore starts with no usable predictor at exactly the moment early dropout is heaviest.

We propose that this transfer failure has a specific, fixable cause. The behavioral signals that predict dropout are mixed with nuisance variation: hardware speed, connection quality, how long a trial's pages are, and individual baseline speed. That mixture differs from one trial to the next. We will learn a dropout representation that is forced to be orthogonal to a separate nuisance representation, meaning the two point in unrelated directions. The nuisance representation is trained on auxiliary prediction tasks whose labels cost nothing to collect, such as device type, page-load latency and trial identity. Only the dropout part is shared across intervention programs, through federated learning, a training scheme in which each program keeps its raw logs and sends only model updates to a coordinating server. The question is whether a new intervention, with a few dozen early participants, can inherit a working dropout predictor from other programs without anyone pooling sensitive telemetry.

Core Claim

Cross-trial dropout prediction fails mainly because the useful signal is confounded with nuisance factors that vary by trial. If nuisance predictors are trained explicitly and kept orthogonal to the dropout predictor, the dropout predictor transfers to an unseen trial with few labels. It should do clearly better than plain parameter sharing, plain federated averaging, and simple per-trial standardization of features.

The claim can be falsified in three ways. Per-trial z-scoring might recover the same gains. The orthogonality penalty might add nothing over ordinary multi-task training with the same auxiliary tasks. A probe might show that the learned dropout representation still encodes device and trial identity.

Relation to Prior Work

The motivating evidence comes from the MindTrails study above. Its authors found passive behavior to be the strongest predictor, with mean Gini importance of 0.029 to 0.045, above demographics and symptom surveys. They also observed the cross-trial failure and named cross-study transfer as unsolved. They pooled all data centrally and did not test any transfer remedy. We take their feature set and their three trials as our testbed, and the transfer remedy is what we add.

The method comes from Exploiting Unrelated Tasks in Multi-Task Learning. That paper adds an orthogonality penalty between the weights for a main task and those for "unrelated" auxiliary tasks. The auxiliary tasks absorb identity-specific variation in facial emotion recognition, which helps with 30 to 50 training samples. We move that idea from faces to interaction telemetry and from a single dataset to a federated, cross-trial setting. In our version the nuisance weights stay local to each program and only the orthogonalized dropout weights are averaged.

Two further results justify sharing parameters at all and set our expectations. In Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding, sharing parameters across domains helped most when target data was scarce, by up to 60% F1 with 200 target sentences. In Boosted multi-task learning, small search markets gained from a shared global model plus market-specific corrections. Both share parameters naively and do nothing about nuisance variation.

The federated health work constrains the design. Privacy-Preserving Federated Depression Detection From Multisource Mobile Health Data and FedMood show that federated averaging over phone telemetry beats isolated local models by 10 to 18 points. They also show it drops from 85.13% to 76.95% accuracy when client data differ in distribution, and they predict mood rather than dropout. FedHealth shares lower layers and then aligns feature statistics locally with CORAL, a step that matches feature means and covariances between source and target. FedMAT learns a per-user attention mask over a shared backbone. Both personalize to each client after the fact. Neither identifies what the nuisance is or removes it from the shared representation, which is the specific difference we test.

Federated Multi-Task Learning (MOCHA) learns per-client models together with a client relationship matrix. Its guarantees need convex losses. Our trials have only 252 to 699 participants, so we deliberately use linear or shallow models where that convexity holds, rather than deep sequence models.

Research Program

Each intervention program $t$ is a federated client, or silo, with a few hundred participants. Each participant has a Session 1 feature vector $x$, a dropout label $y$, and auxiliary nuisance labels $a$: device type, mean page-load time, and trial identity, which can be computed only at a central evaluator. The model has two parts. Shared dropout weights $W_s$ map $x$ to a low-dimensional representation $W_s^\top x$, followed by a dropout classifier. Local nuisance weights $W_n^{(t)}$ map $x$ to predictions of $a$. Each silo minimizes

$$ \mathcal{L}_t = \ell_{\text{drop}}(W_s; x, y) + \lambda\, \ell_{\text{aux}}(W_n^{(t)}; x, a) + \gamma \,\lVert W_s^\top W_n^{(t)} \rVert_F^2 , $$

The terms are defined as follows. $\ell_{\text{drop}}$ is a class-weighted logistic loss for dropout, weighted to handle the retention imbalance the MindTrails authors corrected with SMOTE. $\ell_{\text{aux}}$ is the sum of auxiliary losses. $\lVert\cdot\rVert_F$ is the Frobenius norm, the square root of the sum of squared entries. $\lambda$ and $\gamma$ set how much weight the auxiliary tasks and the orthogonality penalty get.

The server averages only $W_s$ across silos, weighted by silo size as in standard federated averaging. A new trial receives $W_s$, fits its own nuisance weights and a small task-specific head on its first $k$ labeled participants, and predicts dropout for everyone else.

The program has three stages. First, reproduce the MindTrails within-trial results with XGBoost (gradient-boosted decision trees) and quantify the cross-trial gap, which the paper does not report numerically. Second, build the orthogonal model and its ablations, first centrally and then under simulated federation with each trial as a silo. Third, run a mechanism analysis. We train linear probes to predict device type and trial identity from $W_s^\top x$ and test whether low probe accuracy goes with better transfer. This analysis distinguishes "orthogonality removes nuisance" from "orthogonality is just extra regularization."

A secondary stage applies the same recipe to engagement lapse in the Oralytics simulation used by Replicable Bandits for Digital Health Interventions. That paper shows downstream allocation policies must remain smooth to keep post-trial inference valid. The stage asks whether better transferred predictions, fed into a smooth Boltzmann allocation of nudges, change engagement while keeping confidence-interval coverage near nominal.

Baselines

The naive reference is a target-only model: XGBoost trained on the $k$ labeled participants from the new trial, with no outside information. Next to it is source-only transfer, where XGBoost trained on the other two trials is applied unchanged. This reproduces the failure mode the MindTrails paper reports.

The strongest existing within-trial result is the MindTrails XGBoost. It reports F1-macro of 0.917 on Calm Thinking, 0.832 on Managing Anxiety and 0.770 on Future Thinking, each trained on the full trial. These are ceilings for the full-data, single-trial case, not cross-trial numbers. No reported result found on Lacuna for cross-trial transfer F1 on these trials, so we will produce that number ourselves.

The strongest privacy-free competitor is centralized pooling: XGBoost trained on all source trials plus the $k$ target participants, with features z-scored within each trial and trial identity as an input. This is the simple fix a practitioner would try first, and our method must match or beat it to justify the extra machinery.

The strongest federated competitors are three, each run with the same feature set and the same $k$ target labels:

  • FedAvg followed by local fine-tuning of the classifier head.
  • A FedHealth-style variant that freezes the shared layers and applies CORAL alignment locally.
  • A FedMAT-style learned per-silo feature mask over the shared encoder.

Every method gets the same hyperparameter search budget, 50 configurations tuned on source-trial validation folds and never on the target. Results are averaged over 20 random draws of the $k$ target participants.

The key ablations are three:

  • Removing the orthogonality penalty while keeping the auxiliary tasks ($\gamma = 0$), which isolates orthogonality from extra supervision.
  • Removing the auxiliary tasks altogether.
  • Replacing the nuisance labels with shuffled labels, which checks that the benefit depends on the nuisance content and not on the penalty acting as a generic regularizer.

Beating the baselines means two things at $k \le 50$ target participants. Our method must show a higher mean F1-macro than both pooled z-scored XGBoost and the best federated personalization baseline, with paired bootstrap intervals over the 20 draws excluding zero. It must also beat the $\gamma = 0$ ablation by a similar margin.

Why not just…?

Why not just pool the trials centrally and standardize features per trial? This is the most serious threat. The MindTrails trials share one platform and one research group, so pooling is legally plausible there. Per-trial z-scoring removes shifts in the mean and spread of features, which may be most of the nuisance. It cannot remove nuisance that interacts with the dropout signal. One example is page-load latency that predicts dropout on slow devices only because those users are frustrated by the hardware rather than by the intervention. The pooled baseline settles this directly. If it matches our method, we will report that the transfer gap was a scaling artifact and drop the orthogonality claim. Federation remains motivated for programs run by separate developers, where FedMood's authors argue that behavioral telemetry identifies individuals even after obvious personal identifiers are removed. That argument does not apply to MindTrails itself.

Why not just use FedAvg with local fine-tuning, or FedHealth's CORAL alignment? These methods adapt a shared model to each client after training. CORAL matches feature statistics but does not decide which variation is nuisance and which is signal, so it can align away real differences in dropout behavior. FedHealth's reported gain of 5.3 points over a non-personalized global model came on activity recognition with plentiful per-subject data, not on a few dozen labels. Both are run as baselines, and the shuffled-label ablation shows whether naming the nuisance matters beyond generic adaptation.

Why not just train each new trial's model on its own data? At Session 1 a new trial has almost no labels. The MindTrails within-trial scores come from hundreds of participants, and the cold-start evidence from Jaech et al. shows single-domain models degrade sharply at small sample sizes. The target-only curve across $k$ will show exactly where outside information stops helping. If target-only XGBoost catches up by $k = 50$, the practical value of transfer is limited to the first weeks of a launch, and we will say so.

Proposed Experiments

The decisive experiment is leave-one-trial-out on the three MindTrails trials. In each fold two trials are source silos and the third is the target. Methods are compared at $k \in \{10, 25, 50, 100\}$ labeled target participants and with the full target trial. We report F1-macro (to match the prior work), AUROC (area under the curve of true-positive against false-positive rate), and calibration error. The full-target condition is compared against the published within-trial F1 values. Access to the MindTrails logs must be arranged with the trial team, because the file describes them only through the paper.

The mechanism check uses the probes described above. We expect device and trial identity to be decodable from the source-only XGBoost features and from the $\gamma = 0$ representation, but near chance from the orthogonal representation. We also expect the per-fold transfer gain to rise with how far apart the source and target trials are in nuisance statistics.

The federation check compares the centrally trained orthogonal model with its federated version. The gap tells us whether keeping nuisance weights local costs accuracy. The Oralytics stage then reports engagement and confidence-interval coverage under a Boltzmann allocation driven by each predictor.

A clear negative result in any of these is informative. If pooled z-scoring wins, or if probes show no nuisance removal, the field learns that cross-trial attrition transfer needs different inputs rather than better learning methods.