Risk models built on wearable data are almost always evaluated on cleaned-up archives, where every reading that eventually reached the server sits in its correct place. A live deployment sees something different. At the moment a prediction is made, some readings have not yet synced from the phone. Some never will, because the device was off the wrist, on the charger, or broken. Others are corrupted by motion. Models that treat missing data as a clinical signal will read all three as the same thing. The best known of these is GRU-D, a recurrent network (a model that carries a running memory of the sequence so far) that learns how quickly to "forget" a variable once it stops being observed. Because they rely on missingness, these models may look best in retrospective tests and degrade most once deployed.
We propose two things. The first is an evaluation method, an "as-of replay," that rebuilds what a live system would actually have seen at each prediction time, using measurement and arrival timestamps. It extends the shift decomposition of Mind the Performance Gap to streaming wearable data. The second is a small change to the model. A separate module labels every gap by its likely cause, and the recurrent predictor learns a different forgetting rate for each cause. The central question is whether the replay predicts live performance well enough to stand in for it, and whether typing the gaps closes the part of the gap that comes from data arriving late.
Core Claim
For wearable-based clinical prediction, a large share of the retrospective-prospective gap comes from a shift in how data goes missing, not from changes in the patient population. Archives backfill late-arriving data. They also often exclude participants who stopped wearing the device. Live streams contain neither correction. We expect two consequences.
First, the shift is roughly predictable from arrival logs alone, so an as-of replay of archived data should estimate live discrimination and calibration far better than the standard archive evaluation. Second, models that learn from missingness without knowing why data is missing should show the largest drop. Separating gap cause from the recurrent update should recover most of it.
The new element is not another missing-data recurrent network. It is treating the cause of a gap as an explicit, swappable input. The same timestamp-faithful replay then feeds every model, so every model is compared under conditions that match deployment.
Why This Gap Exists in the Literature
Recurrent methods for incomplete series have been tested only where the missingness process is frozen.
- GRU-D, from Recurrent Neural Networks for Multivariate Time Series with Missing Values, adds learned decay of inputs and memory over elapsed time. It reaches 0.8527 AUC for MIMIC-III mortality. It assumes the missingness seen in training is the missingness seen at deployment.
- Interpolation-Prediction Networks, from Modeling Irregularly Sampled Clinical Time Series, separate the interpolation step from a GRU predictor and report 0.853 AUC against 0.835 for GRU-D. This shows that decoupling helps. Their interpolator still treats every gap identically.
- The Bayesian recurrent model of Bayesian Recurrent Framework for Missing Data Imputation and Prediction with Clinical Time Series adds calibrated uncertainty, but tests only on artificially dropped values that are missing at random.
- Opportunistic teacher forcing, from Training Recurrent Neural Networks with Inherent Missing Data for Wearable Device Applications, lets a network run on its own predictions when inputs are absent. It was tested only on a Lorenz system with 80% random missingness, and the authors flag long contiguous dropouts as untested.
On the transformer side, LSM-2 handles naturally occurring sensor gaps with one shared learnable mask token. Its accuracy drops 57% (versus 71% for its predecessor) when the accelerometer is removed. That mask token, too, is blind to why data is absent.
These papers supply our methods. None of them measures behaviour under live missingness.
Two other lines of work motivate the proposal. The wearability literature shows that wearable gaps are bursty and caused by behaviour, not random:
- The COPD smartwatch cohort in Feasibility of Using a Smartwatch to Intensively Monitor Patients With Chronic Obstructive Pulmonary Disease lost an average of 13 days of heart rate and 17 days of activity per participant to sync failures. Only 57% of participants completed the study.
- Removal is driven by comfort and restricted movement, especially during sleep (The role of comfort, personality, and intention in smartwatch usage during sleep), and by tethered probes (Wearability Testing of Ambulatory Vital Sign Monitoring Devices).
The prospective-validation literature shows the gap can be decomposed. Ötles et al. used matched snapshots to split a 0.011 AUROC drop (0.778 retrospective, 0.767 prospective) into a 0.016 loss from infrastructure and a 0.005 gain from temporal change. They also found that the live pipeline silently failed on 10 of 356 days. Their study used static tabular models on inpatient records, with no streams and no recurrent networks. Our design transfers their logic to a setting where infrastructure shift takes the specific form of arrival latency and non-wear.
Research Program
The availability module
The first component is a gap-typing module with a fixed interface. For each channel and time step, it takes raw sensor values, measurement timestamps, server arrival timestamps, and the query time. It outputs a value, an elapsed-time-since-last-observation, and a gap type. The gap types are:
- observed;
- pending (measured but not yet arrived as of the query time);
- non-wear (inferred from flat accelerometry with absent heart rate);
- device-off or charging (no packets at all, battery trace where available);
- low signal quality.
The rule-based version uses thresholds. The learned version is a small classifier trained on the subset of days where self-report or charging logs give ground truth.
The predictor
The second component is a GRU with decay of the GRU-D kind. Its forgetting rates for inputs and memory are indexed by gap type, so "pending" and "non-wear" can learn different behaviour. During long device-off bursts it optionally runs autonomously, in the manner of opportunistic teacher forcing.
Why the interface matters
The modular boundary does real work here; it is not packaging. The same availability module feeds every competitor: a GRU-D, a LightGBM on daily summary features, and a masked transformer. It also feeds both the retrospective replay and the live shadow system. Differences in results then cannot come from mismatched preprocessing. This is the variable-isolation argument made for evaluation toolkits such as Universal Causal Evaluation Engine and UniTS, applied to the one variable that matters most in this setting.
The as-of replay
The third component is the replay harness. For each prediction time $t$, it constructs the input from only those readings whose arrival timestamp is at or before $t$. This mirrors what a live server would hold.
Shift decomposition
Following Ötles et al., we evaluate a model trained on an earlier period on four snapshots of a later period:
- the archive extracted long after the fact (backfilled);
- the as-of replay of that same period;
- the live shadow-mode stream from that period;
- the earlier-period archive, used for reference.
The differences between snapshots separate three effects. Temporal shift is the earlier archive versus the later archive. Latency shift is the later archive versus the later replay. Residual live-pipeline effects are the later replay versus the live stream.
Settings
The main setting is post-discharge 30-day readmission or death in older adults wearing consumer trackers. This is the task of Optimizing Temporal Windows for Wearable-Augmented Post-Discharge Risk Prediction, which used the PREDICT and MOVE IT trial cohorts (215 patients), and we reuse its windows: 3 to 10 day look-back and a 10-day horizon.
The decomposition depends on having arrival timestamps. If the trial's device-vendor logs lack them, the replay will be built from a new shadow-mode cohort. In that cohort, a system scores discharged patients daily without showing scores to clinicians and logs every arrival. Shadow mode measures live model behaviour, not effects on clinicians. We treat clinician-facing deployment as later work and state this explicitly.
The COPD smartwatch logs serve only to fit realistic burst-length and latency distributions for stress tests, not as a prediction benchmark, given its 24 exacerbations. MIMIC-III and PhysioNet 2012 serve as a reproduction check, to confirm our GRU-D implementation matches published numbers before we trust it elsewhere.
Baselines
Naive reference. A GRU fed forward-filled values with a missingness mask, plus the "do nothing" model that predicts the cohort base rate. These set the floor. They also test whether any missingness modelling helps once missingness is honest. In the GRU-D paper, a GRU with mean imputation scored 0.8252 against 0.8527 for GRU-D on MIMIC-III.
Strongest existing method for this task. LightGBM on daily step, distance and sleep summaries, as in the Penn methods study. That study reports a mean ROC-AUC of 0.82 across configurations, a peak of 0.87 at a 10-day horizon, and good calibration (Hosmer-Lemeshow p = 0.96). These numbers come from offline analysis of the archive. We run it on the identical availability-module output, with the same patient splits and the same hyperparameter search budget (an equal number of trials of random search) as every neural model. "Beating it" means one of two outcomes:
- higher AUROC on the as-of replay and live snapshots, with confidence intervals from patient-level bootstrap that exclude zero difference; or
- equal AUROC with a significantly smaller replay-to-archive gap and better calibration slope on the live stream.
Matching 0.87 on the backfilled archive alone would not count.
Strongest missingness-aware recurrent method. GRU-D with untyped gaps, trained on the same inputs. This is the model our hypothesis predicts will shift most.
Strongest non-recurrent missingness method. A masked 1D transformer trained from scratch on the same data, with LSM-2-style shared mask tokens for natural gaps. Whether pretrained LSM-2 weights can be used is unknown to us, so the fair comparison is architecture-matched and trained on the same data. There is no reported result found on Lacuna for any model on post-discharge readmission under as-of replay.
Key ablations. We isolate the proposal's key ingredient three ways:
- the typed GRU with all gap types collapsed into one, which recovers GRU-D;
- the untyped GRU-D trained on replay-style inputs with simulated latency;
- the typed model with the rule-based versus learned availability module.
Why not just…?
Why not just use daily summaries and gradient-boosted trees, since three days of look-back already works? This is the most serious threat. The Penn study found look-back length barely mattered and reached 0.87 with LightGBM. A recurrent model may add nothing to discrimination. The proposal's main contribution, the as-of replay and decomposition, does not depend on recurrent networks winning, so this baseline is run inside the same harness. If LightGBM matches the typed GRU on live AUROC and calibration, we will report that the recurrent component is unnecessary. The remaining question is then whether gap typing helps LightGBM too, which the interface lets us test directly by adding gap-type features.
Why not just train GRU-D on as-of replay data instead of adding gap types? This augmentation may close the latency gap by itself, because the model would see realistic gaps during training. It is one of our ablations. Gap typing is justified only if it beats latency-augmented GRU-D on the live snapshot. The most likely place for that is long device-off bursts, where "pending" and "removed" call for opposite behaviour: wait, versus distrust stale memory. If the two tie, the simpler augmentation is the recommendation.
Why not just use a large wearable foundation model with masking? LSM-2 shows that masking handles natural gaps well at scale. But its single mask token cannot tell a sync delay from a removal, and it was evaluated only on retrospective splits. The architecture-matched masked transformer baseline, with and without gap-type embeddings added to its tokens, tests whether the benefit belongs to typing rather than to recurrence. A positive result for typed transformers would still support the core claim.
Why not just read the Ötles decomposition off the archive and skip the shadow deployment? The as-of replay is exactly that shortcut. Whether it is trustworthy is an empirical question that only the live stream can answer, because the 10 silent pipeline-failure days in Ötles et al. are the kind of event no replay reproduces. The live snapshot is therefore the test of the replay, not a redundant extra.
Decisive Checks
Replay fidelity
The first check is whether the as-of replay predicts live performance. We compare the absolute error between replay and live AUROC, and between replay and live calibration slope, with the same errors for the standard backfilled archive. Success is the replay being closer to live on both, across all models.
Rank reversal
The second check is whether models reorder between evaluation settings. We test whether untyped GRU-D beats the forward-fill GRU on the backfilled archive but loses its advantage on the replay and live stream, while the typed model keeps it. Observing this reversal would confirm the mechanism. Not observing it would mean missingness shift is not the dominant effect, and the decomposition would show where the gap actually comes from.
Stress tests
Burst-length stress tests, using COPD-fitted dropout distributions, check whether autonomous running helps beyond multi-day gaps. Opportunistic teacher forcing has never been tested on such gaps.
Subgroups, burden and uncertainty
We report performance stratified by wear compliance and by sex. Retention bias toward certain users, such as the 36% female enrolment in the COPD cohort, would otherwise make low-compliance patients invisible in the archive. We also report alert burden per patient-week at a fixed sensitivity, so that gains in AUROC are not bought with alert fatigue. Finally, we test whether predictive variance, estimated by Monte Carlo dropout, rises with data staleness. If it does, the system could withhold scores during long gaps, following the variance-error relationship of the Bayesian recurrent framework.
References
- Mind the Performance Gap: Examining Dataset Shift During Prospective Validation
- Recurrent Neural Networks for Multivariate Time Series with Missing Values
- Modeling Irregularly Sampled Clinical Time Series
- Bayesian Recurrent Framework for Missing Data Imputation and Prediction with Clinical Time Series
- Training Recurrent Neural Networks with Inherent Missing Data for Wearable Device Applications (Student Abstract)
- LSM-2: Learning from Incomplete Wearable Sensor Data
- Feasibility of Using a Smartwatch to Intensively Monitor Patients With Chronic Obstructive Pulmonary Disease: Prospective Cohort Study
- The role of comfort, personality, and intention in smartwatch usage during sleep
- Wearability Testing of Ambulatory Vital Sign Monitoring Devices: Prospective Observational Cohort Study
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- UniTS: A Universal Time Series Analysis Framework Powered by Self-Supervised Representation Learning
- Optimizing Temporal Windows for Wearable-Augmented Post-Discharge Risk Prediction: A Methods Study