Freezing of gait is a sudden, brief inability to step that affects many people with Parkinson's disease. It is a main cause of falls, and clinic visits rarely capture it, because a neurologist sees the patient a few times a year and the freeze may never happen in the room. Wearable motion sensors can watch for it all day. The datasets used to build such detectors, however, contain about ten to two dozen patients. Freezing windows are a small minority of the recordings, and the start and end of each episode is marked by hand with real uncertainty. The usual fix for the imbalance is to repeat the rare freezing windows until the classes are even, sometimes with small time shifts. With so few patients, this means the network sees the same handful of episodes, with the same hard and often imprecise labels, many times over. Reports disagree on whether that causes memorization: full 1:1 balancing is recommended for image classifiers, while a 0.5:1 ratio worked better on small health-text tasks.
We propose to keep the oversampling but change its labels. Each repeated or time-shifted freezing window gets a soft label, meaning a probability rather than a 0/1 tag. That probability comes from a "teacher" network that never saw the patient the window came from. The small on-sensor detector, the "student," is then trained on this rebalanced and relabeled stream. Our claim is that the extra minority windows then add information instead of duplicates. The teacher's out-of-patient opinion softens the label noise at episode boundaries and limits how much one patient's idiosyncratic freezes can dominate training. We expect a compact student trained this way to match or beat its larger teacher under leave-one-subject-out testing, meaning the model is always evaluated on a patient it has never seen.
Core Claim
The claim is an interaction, and it can be falsified. With hard labels, heavier oversampling should eventually hurt in tiny cohorts. With cross-fitted teacher soft labels, that penalty should shrink or vanish, so full 1:1 balancing becomes safe and the best choice. "Cross-fitted" means each training patient's windows are labeled by a teacher trained without that patient. If the result holds, it resolves the disagreement between A systematic study of the class imbalance problem in convolutional neural networks and CASIA@SMM4H'22 for clinical time series. The disagreement would be traced to label quality on the repeated samples rather than to oversampling itself. A second claim is that the benefit comes from the teacher's patient-held-out knowledge, not from softness alone. A hand-built soft label that ramps down near annotated boundaries should therefore fall short.
Where This Sits
The proposal draws on three groups of prior work.
The first group motivates the gap. Freezing-of-gait detectors on small cohorts handle imbalance with weighted losses or simple shifting, and none uses distillation:
- Deep Learning for Detecting Freezing of Gait Episodes in Parkinson's Disease Based on Accelerometers uses a 1D ConvNet with weighted hinge loss on 15 home-recorded patients.
- Increasing Robustness in the Detection of Freezing of Gait in Parkinson's Disease uses speech-style features on Daphnet. It also shows that some patients' freezes lack the classic 3–8 Hz trembling signature, which is exactly the kind of atypical episode that hard labels on repeated windows will overfit.
- FM-FoG reaches high accuracy with a masked-reconstruction foundation model. It needs a gatekeeper network to keep phone battery drain tolerable, so it is not a micro-power detector.
The second group constrains how oversampling can be done. Synthesis that scrambles temporal structure is risky. Shuffling decomposed signal components helped connectivity-based models but hurt a raw-waveform model in Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease. FreMGP evolves synthetic samples in the frequency domain and outperforms SMOTE on single-channel UCR data, but it is untested on multi-axis sensor streams. Our oversampler therefore uses only real, temporally intact crops: windows cut at jittered offsets around and across annotated episodes. The new ingredient is how those crops are labeled, not how they are generated.
The third group supplies the mechanism and its limits. A Knowledge Distillation Approach for Sepsis Outcome Prediction shows that a structured clinical time-series student can beat its LSTM teacher on secondary tasks. Its reported AUROCs are 0.819 against 0.817 for dialysis and 0.638 against 0.601 for ventilation. That work uses no oversampling and a fixed ICU window. Ranking Distillation combines ground-truth and teacher supervision to stop a compact student from overfitting sparse labels. We use the same dual supervision, applied to the minority windows.
Two theory papers shape the design:
- On the Mechanisms of Weak-to-Strong Generalization proves that a student cannot beat an over-regularized teacher in the ridge setting. We take this as a prediction to test rather than a guarantee for deep networks. Accordingly, the teacher is trained lightly regularized, and teacher regularization strength is swept explicitly.
- Optimal Unconstrained Self-Distillation in Ridge Regression shows that letting the hard/soft mixing weight leave $[0,1]$ can strictly help, and that distilling on the same training inputs beats fresh inputs. We borrow the unconstrained mixing weight. We expect the "same inputs" result not to transfer when the teacher is an overparameterized network that memorizes its training data, because there an in-sample teacher's soft labels collapse back to the hard labels. That contrast is one of our checks.
Research Program
Each training window $x$ drawn by the oversampler from training patient $j$ receives the target
$$y_\xi(x) = (1-\xi)\,y_{\text{hard}}(x) + \xi\,p_{T}^{(-j)}(x).$$
The symbols are:
- $y_{\text{hard}}$ is the annotated label of the window.
- $p_T^{(-j)}$ is the freezing probability from a teacher trained on all training patients except $j$ (and, of course, except the held-out test patient).
- $\xi$ is a mixing weight tuned by nested leave-one-subject-out validation, with negative values and values above one allowed, following Dang et al.
The student minimizes cross-entropy against $y_\xi$. The oversampling ratio $r$ sets how many minority-region crops are drawn per majority window. Non-freezing windows keep their hard labels in the main condition, and a secondary condition relabels them too.
The work has four parts:
- Teachers. We build cross-fitted teachers, using CNN-transformer models trained with weak regularization. For Daphnet's ten patients this means at most ninety small teacher fits per full experiment, which is cheap; a K-fold approximation is the fallback for larger cohorts.
- Oversampler. We build the jittered-crop oversampler with ratio $r \in \{0, 0.25, 0.5, 1\}$.
- Student. We train a student held to a fixed microcontroller-scale budget. Parameter count and multiply-accumulates are fixed before any results are seen and reported alongside accuracy.
- Analysis. We measure where the teacher's soft labels deviate most from the annotations: near boundaries, on atypical non-trembling freezes, and on the patients who contribute the most episodes.
Baselines
The naive reference is the same student trained on the natural class distribution with plain cross-entropy and a tuned decision threshold. This is the floor that any imbalance method must clear.
Strongest simple imbalance method. This is random oversampling with hard labels at $r=1$ and at $r=0.5$, the two ratios recommended by Buda et al. and Fu et al.. We also run class-weighted loss in the style of the 1D ConvNet work. All three use the identical student, crops, epochs and nested-LOSO tuning budget.
Strongest synthetic oversampler. This is FreMGP, applied per sensor axis with hard labels. No reported result was found on Lacuna for FreMGP on any freezing-of-gait dataset, so we will report our own run and disclose its added compute.
Strongest published detectors. These set external reference points rather than budget-matched competitors:
- On Daphnet under leave-one-subject-out, San-Segundo et al. report an equal error rate of 12.5% and specificity of 0.75 at 0.95 sensitivity.
- On the 15-patient home dataset, the 1D ConvNet paper reports 88.6% sensitivity and 78% specificity.
- On VCU FoG-IMU, FM-FoG reports 98.5% F1 on unseen patients, against 90.9% for Gait-Guard.
- No reported result was found on Lacuna for tDCS FOG or DeFOG.
Beating the baselines. Success means the budget-matched student beats the best hard-label oversampling configuration on equal error rate and on specificity at 0.95 sensitivity, with paired per-patient tests across folds. On Daphnet it must also reach or pass the 12.5% figure. On VCU we do not expect a micro-budget student to pass 98.5% F1. Closing most of the gap to FM-FoG at a small fraction of its compute is the deployment claim.
Ablations. These isolate the key ingredient:
- uniform label smoothing on oversampled windows;
- a hand-built boundary ramp that sets soft labels by distance to the annotated onset and offset;
- an in-sample teacher, trained with patient $j$ included;
- ordinary distillation with no oversampling;
- a sweep of teacher regularization strength.
Why not just…?
Why not just use a class-weighted loss? Class weighting multiplies the gradient of the same few hard-labeled freezing windows. It adds no new window positions and does nothing about boundary label noise, so we expect it to behave like hard-label oversampling at high ratios. It is included as a baseline at the same budget. If it matches teacher-labeled oversampling, the proposal's central claim fails.
Why not just smooth the labels near episode boundaries by hand? This is the most serious threat. Much of the boundary noise may be captured by a simple ramp, and if the ramp matches the teacher, the teacher is wasted effort. Our bet is that the teacher's advantage shows up where the ramp cannot help: atypical freezes without the trembling signature, which San-Segundo et al. identify as the main cross-patient failure, and windows from the patients who dominate the episode count. We will report results separately for these subsets. If the ramp ties the teacher overall and on these subsets, we will report the method as "boundary-aware soft oversampling" and drop the distillation claim.
Why not just deploy the foundation model or the teacher itself? FM-FoG needed a gatekeeper even on a phone, and a sensor-resident detector has far less room. Beyond compute, our hypothesis is that the student can outperform the teacher on held-out patients, as the sepsis work found on two of three tasks. We test this directly by reporting teacher and student on the same folds.
Why not just synthesize new freezing samples? Synthesis that rearranges signal components degraded a raw-waveform model in the EEG augmentation study. FreMGP is untested on multi-axis motion data and is costly to run. Including FreMGP as a baseline lets us check whether its synthetic diversity beats real jittered crops with better labels. Combining FreMGP samples with teacher labels is a natural follow-up if both help.
Proposed Experiments
The main experiment is a full grid over oversampling ratio $r$ and label source (hard, uniform smoothing, boundary ramp, in-sample teacher, cross-fitted teacher). It runs under leave-one-subject-out on Daphnet and VCU FoG-IMU, and on the home dataset of Rodríguez et al. if access can be obtained. tDCS FOG and DeFOG serve as cross-dataset transfer tests.
We report:
- equal error rate;
- specificity at 0.95 sensitivity;
- F1 and geometric mean;
- event-level detection latency;
- calibration error;
- worst-patient performance;
- student compute.
The decisive check is the interaction. Hard-label performance should peak below $r=1$, while cross-fitted-teacher performance should rise monotonically to $r=1$. If the curves are parallel, the claim is false.
Two further checks probe the mechanism. First, student gains should shrink as teacher regularization increases, as the over-regularized-teacher bound predicts. Second, an in-sample teacher should give roughly hard-label results, which tells us whether patient-held-out labeling, rather than teacher capacity, is what matters.
The final analysis is a per-patient attribution. We ask whether the improvement concentrates on patients with atypical freezes and on patients whose episodes are rare in the training folds. That is where the account of intermittent, heterogeneous clinical events says the gain should appear.
References
- A systematic study of the class imbalance problem in convolutional neural networks
- CASIA@SMM4H'22: A Uniform Health Information Mining System for Multilingual Social Media Texts
- Deep Learning for Detecting Freezing of Gait Episodes in Parkinson's Disease Based on Accelerometers
- Increasing Robustness in the Detection of Freezing of Gait in Parkinson’s Disease
- FM-FoG: A Real-Time Foundation Model-based Wearable System for Freezing-of-Gait Mitigation
- Assessing the Potential of Data Augmentation in EEG Functional Connectivity for Early Detection of Alzheimer's Disease
- Contrastive Representation-Guided Genetic Minority Oversampling for Imbalanced Time-Series Classification
- A Knowledge Distillation Approach for Sepsis Outcome Prediction from Multivariate Clinical Time Series
- Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
- Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning