Wearable sensors can now flag "freezing of gait," the sudden, seconds-long episodes in Parkinson's disease when a patient's feet stop moving despite the intent to walk. Published detectors look nearly solved. One reports a 98.5% F1-score on unseen patients. But these numbers are counted over short fixed time windows recorded during in-clinic obstacle courses designed to provoke freezing. In that setting, freezing is common and every patient is observed for minutes, not weeks. A clinician deploying such a device would ask different questions. How many real episodes did it miss, especially the brief ones? How late did it fire? How many false alarms did it produce per hour of ordinary walking at home, where freezing is far rarer? We propose a patient-level, event-based scoring protocol that answers these questions and lets the assumed real-world frequency of freezing be varied. We then use it to test whether the class-imbalance fixes that work in medical imaging carry over to wearable data, and whether they hold up once a detector is scored this way.
The core bet is that the field's bottleneck is now measurement rather than model capacity. Window-level F1 on provoked clinic sessions can rank detectors differently from what matters for deployment. With only 10 to 23 patients per public dataset, many reported gaps between methods may fall inside patient-to-patient noise. If that is true, the imbalance machinery imported from imaging will look either more or less useful than it really is, depending on which metric is used.
Core Claim
We claim three testable things. First, under leave-one-subject-out evaluation, ranking freezing detectors by window-level F1 disagrees with ranking them by event recall at a fixed false-alarm budget per walking hour, and the disagreement grows as the assumed real-world frequency of freezing drops. Second, with these cohort sizes, a substantial share of published between-method gaps are not resolvable once confidence intervals are computed by resampling patients rather than windows. Third, a specific imaging remedy gains event recall on short episodes that simply moving the decision threshold of an ordinary model cannot reproduce. That remedy is decoupled classifier retraining: learn the feature encoder on the natural, skewed data, then refit only the final layer on class-balanced samples. If the third claim fails, that is itself a useful negative result. It would mean the gains seen for imaging do not transfer to binary temporal event detection, and that the real lever is the choice of operating threshold.
Where This Sits
Two papers motivate the scoring protocol. Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring scores language-model summaries of dementia home-monitoring data by abnormality recall, duration recall, coverage and hallucinated events. It shows that fluent summaries often silently omit dangerous episodes: zero-shot recall was 0.0–2.1%. Its events come from fixed thresholds applied to coarse vital signs, and it scores text, not detectors. We borrow its event-level vocabulary but apply it directly to the outputs of sensor classifiers, against physiotherapist-annotated events. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation built RadCliQ, a composite metric fit to radiologist error counts (Kendall-tau b 0.522). It also showed that apparent progress shrinks to a 5.5% gain over random retrieval once scoring is clinically grounded. That is the pattern we expect to find in freezing detection. We do not fit a composite to expert ratings, because no such ratings exist in the file's datasets. We instead report the separate clinical quantities explicitly.
The detectors we evaluate set the bar. FM-FoG pretrains a transformer by masked reconstruction of motion signals and reports window F1 only. Increasing Robustness in the Detection of Freezing of Gait in Parkinson's Disease uses hand-built frequency features with context windows on Daphnet and reports equal error rates. Neither measures onset delay, short-episode recall or false alarms per hour. The imbalance method comes from Long-Tailed Classification of Thorax Diseases on Chest X-Ray. There, classifier retraining raised tail accuracy from 1.7% to 30.0% on NIH-CXR-LT. The same paper found that MixUp, which blends pairs of training inputs, erases subtle lesions. That finding constrains us. Balanced-MixUp reports the opposite result on fundus images. Rather than adding a third mixing variant to this unresolved dispute, we leave mixing out. The small-sample framing draws on two sources. Modeling multiple sclerosis using mobile and wearable sensor data showed that daily wearable features are too unreliable without aggregation. MONICA explicitly noted that the field lacks metrics that trade head-class against tail-class performance.
Research Program
The scoring protocol works on contiguous predicted episodes, not windows. A predicted episode counts as a hit if it overlaps an annotated freeze within a tolerance. From hits we compute four quantities: event recall, stratified by episode duration; median onset delay; the fraction of each freeze's duration that the detector covers; and false alarms per hour of non-freezing walking, kept separate from false alarms during turns. Because in-clinic recordings over-represent freezing, we also report precision at an assumed deployment frequency $\pi$, using
$$P(\pi) = \frac{R\,\pi}{R\,\pi + F\,(1-\pi)},$$
where $R$ is event recall and $F$ is the false-alarm rate per unit of non-freezing time, both measured in the clinic. The formula is Bayes' rule with clinic-measured rates held fixed. The file gives no free-living frequency of freezing, so we treat $\pi$ as a parameter to sweep rather than a known constant. All confidence intervals come from resampling patients, so that one patient with many episodes cannot dominate. From the same resampling we report the smallest between-method difference each dataset can resolve.
The modelling study compares four routes to a detector, all built on one shared self-supervised encoder in the style of FM-FoG: end-to-end fine-tuning; end-to-end fine-tuning with focal loss; end-to-end fine-tuning with the threshold tuned to a false-alarm budget; and decoupled head retraining. On WearGait-PD we also run a multi-class version (walking, turning, standing, stagger, freeze), where freeze is the rarest of several classes. The imaging analogy is closest in this setting, so it is where decoupling should matter most.
Baselines
The do-nothing reference is the end-to-end fine-tuned encoder at its default 0.5 threshold, scored by window F1. This is how the field currently reports results. Every other number is read against it.
The strongest existing system is FM-FoG, with masked-reconstruction pretraining and end-to-end fine-tuning. It reports 98.5% F1 on unseen patients on the VCU FoG-IMU cohort, against 90.9% for Gait-Guard (FM-FoG). On Daphnet, the strongest reported number is an equal error rate of 12.5%, and specificity of 0.75 at sensitivity 0.95, from Increasing Robustness. The file contains no Daphnet-only figure for FM-FoG. We reimplement the FM-FoG-style backbone and use identical leave-one-subject-out splits, the same pretraining data, and the same number of tuning trials per arm, selected on inner held-out patients. The VCU cohort may not be publicly available. If so, our headline comparisons use Daphnet and WearGait-PD, and the VCU number serves only as a reference. On WearGait-PD the file reports no predictive result, so our numbers there are first results.
The key ablation is threshold-tuned end-to-end training against decoupled retraining at the same false-alarm budget. This isolates whether retraining the head changes the direction of the decision boundary or only its offset. Beating the baselines means statistically higher event recall on episodes shorter than the median duration, at a matched false-alarm rate per hour, with confidence intervals from patient resampling that exclude zero. A gain in window F1 alone does not count.
Why not just…?
Why not just report precision-recall area or an event-level F1 from sound-event detection? This is the most serious threat to the metric half of the proposal. Most of our protocol is event-level scoring that other fields already use. What we add is fixing a false-alarm budget per walking hour, sweeping $\pi$, and resampling patients, all on freezing data. We test directly whether these additions matter. If precision-recall area at a reweighted frequency ranks methods the same way our protocol does (rank correlation near 1 across all arms and datasets), we will report precision-recall area as sufficient and drop our extra quantities.
Why not just move the threshold instead of retraining the head? In binary detection, retraining a linear head on balanced samples may only shift its bias, which is exactly what moving the threshold does. The imaging evidence for retraining comes from 20-class problems (Holste et al.). This alternative is a real threat, which is why matched-budget threshold tuning is our central ablation. We expect retraining to help mainly in the multi-class WearGait-PD setting. If it does not help there either, we will say so.
Why not just use focal loss? On chest X-rays, reweighted losses such as focal loss trailed decoupled retraining (Holste et al.). On ISIC-2019-LT, simple reweighting struggled under extreme imbalance (MONICA). None of this evidence comes from temporal data, so focal loss is a full arm of the study, tuned under the same budget as the others.
Decisive Checks
The study succeeds on the metric side if, on Daphnet and WearGait-PD, the ordering of methods by window F1 differs from their ordering by event recall at a fixed false-alarm budget for at least one plausible low value of $\pi$, and that reversal survives patient resampling. It also succeeds there if resampling shows that the 10-patient Daphnet cohort cannot resolve gaps of the size the literature reports. On the method side, decoupled retraining must beat the threshold-tuned baseline on short episodes at a matched false-alarm budget. A null result on that last check is still a clear outcome. It would tell practitioners to tune operating points and stop importing rebalancing schemes from imaging.
References
- Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring
- Evaluating progress in automatic chest X-ray radiology report generation
- FM-FoG: A Real-Time Foundation Model-based Wearable System for Freezing-of-Gait Mitigation
- Increasing Robustness in the Detection of Freezing of Gait in Parkinson’s Disease
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
- Balanced-MixUp for Highly Imbalanced Medical Image Classification
- Modeling multiple sclerosis using mobile and wearable sensor data
- MONICA: Benchmarking on Long-tailed Medical Image Classification