Clinicians score Parkinson's disease by watching a patient walk or tap their fingers and assigning an integer from 0 to 4. Two experts watching the same patient disagree up to 53% of the time. The datasets used to automate this scoring are tiny: tens to a few hundred patients, with the severe scores almost absent. This note proposes training a small scoring model not on the final predictions of a large "teacher" model, but on a short sequence of the teacher's predictions taken from different points in its training. This is called progressive distillation. The bet is that partway through training, the teacher's predictions still encode the simple, population-level motor signals that clinicians actually agree on, chiefly movement speed. By the end of training, the teacher has buried those signals under memorized, rater-specific label noise. A student that sees the intermediate teacher first should therefore need fewer patients, generalize better to new clinics, and give probabilities that reflect where human raters genuinely disagree.
The opportunity comes from two findings that have never been put together. In the distillation literature, Progressive distillation induces an implicit curriculum shows that a teacher passes through a phase where it relies on simple features that later vanish. Distilling across that phase cuts the samples a student needs from exponential to polynomial on a synthetic task. In the clinical literature, Computer vision uncovers three fundamental dimensions of levodopa-responsive motor improvement in Parkinson's disease shows that clinical scores compress three separate kinematic dimensions: speed, cycle-to-cycle consistency, and timing. Small-sample learning of a compressed, noisy label is exactly the regime where an implicit curriculum should matter most. That regime is also exactly where the distillation literature has never looked.
Core Claim
On subjective ordinal clinical scores learned from small cohorts, a student distilled first from an intermediate teacher checkpoint and then from the final teacher will beat the alternatives on three measures: ordinal agreement with clinician consensus, cross-site transfer, and recall of rare severe classes. The comparison is against direct training, label smoothing, one-shot distillation from either checkpoint alone, and checkpoint ensembles.
We also make a mechanistic claim that can be checked independently of accuracy. The intermediate checkpoint is the one where simple kinematic quantities are most linearly decodable from the teacher's internal features. By the final checkpoint, the teacher's confidence on rater-ambiguous cases has outrun the actual rater disagreement.
The main lever is sample efficiency and robustness to label noise, not compression. Compression is a secondary benefit we report. We do not rest the case on it. On the sensor benchmark, a compact 1D convolutional network already reports 98.7% detection accuracy (Deep 1D-Convnet for accurate Parkinson disease detection and severity prediction from gait). That is higher than the 95.2% of the heavier Transformer (Transformers for 1D Signals in Parkinson's Disease Detection from Gait). "Shrink the big gait model" is therefore not an obviously open problem.
Research Program
The main testbed is CARE-PD (CARE-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson's Disease Gait Assessment). It contains 8,477 walking segments from 362 subjects across 8 clinical centers. Each segment is a standardized 3D body-mesh sequence labeled with a clinician's UPDRS-gait score from 0 to 3. Its multiple sites let us test transfer by holding out one clinic at a time, and score 3 is rare.
The teacher is a motion Transformer of the MotionAGFormer type that CARE-PD already benchmarks. The student is a small temporal convolutional network over joint trajectories. We run the method twice: once with a student of the same size as the teacher, which isolates the curriculum effect from compression, and once with a student roughly ten times smaller.
During teacher training we save checkpoints densely. At each checkpoint we freeze the features and fit linear probes for handcrafted gait summaries computed directly from the meshes: walking speed, stride-time variability, and cadence. These are rough gait analogues of the speed, consistency, and timing dimensions of Lange et al. The probe curves locate the phase where simple features peak, and test whether any of them decay later, which is the Panigrahi signature. We also track the teacher's training-loss curve for the sharp drop that the Panigrahi paper associates with this phase.
The selected checkpoint is the one where probe accuracy is jointly highest. Selection uses only training-fold data. The student is then trained for a fixed fraction of its budget on that checkpoint's softened predictions, and for the rest on the final teacher's predictions. This is the 2-shot schedule that Panigrahi et al. found captures most of the benefit. The cross-entropy loss on the clinician's label is kept throughout.
Treating the transition as a sharp switch versus a gradual blend is one hyperparameter, tuned on validation. Continuation-style smoothing (Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization), which anneals a hinge margin so the student ignores small logit differences early, appears only as a baseline. It is not part of the method.
The second testbed is the PhysioNet foot-pressure gait data. Its 18 force-sensor channels carry UPDRS and Hoehn–Yahr labels. Here no mesh-based kinematic decomposition exists, so probes use simple force statistics: heel and toe peak force and stance time. This tests whether the method works when the clinical sub-dimensions are less well defined.
The third testbed, TULIP (TULIP: Multi-Camera 3D Precision Assessment of Parkinson's Disease), is too small for training a deep teacher, with 15 subjects. It is the only dataset in the file with three independent expert ratings per item. We use it strictly as an exploratory check on whether student confidence tracks real rater spread, using features from the CARE-PD-trained models.
Baselines
The naive reference is the student architecture trained directly on clinician labels with cross-entropy, at the same compute and tuning budget as every other arm. That budget is a fixed number of epochs and an equal number of hyperparameter trials, all on identical leave-one-site-out folds. If progressive distillation cannot beat this, nothing else matters.
The clinical-practice baseline is the same student with fixed label smoothing. This follows Towards Automated and Marker-Less Parkinson Disease Assessment: Predicting UPDRS Scores Using Sit-Stand Videos, whose label-smoothed ensembles reached F1 of 0.75 on bradykinesia and 0.78 on posture and gait, against 0.50 and 0.45 for human video raters. We also run an ordinal variant where smoothing mass falls off with distance from the true score, because uniform smoothing is an unfair strawman for an ordinal scale.
The distillation baselines come in two families, each tuned with the same trial budget as our method. The first is one-shot distillation, run separately from the final teacher, from the early-stopped teacher (best validation loss), and from the probe-selected intermediate checkpoint alone. The second family contains multi-stage alternatives. EEKD, from Learn From the Past: Experience Ensemble Knowledge Distillation, distills from an attention-weighted ensemble of uniformly spaced teacher checkpoints. It reported 72.91% versus 70.67% for vanilla distillation on CIFAR-100. It is the strongest existing competitor because it uses the same raw material, the teacher's past, without ordering it into a schedule. The others are Continuation KD and, for the small-student condition only, a teacher-assistant chain from Improved Knowledge Distillation via Teacher Assistant.
Two ablations isolate the key ingredient. One replaces probe-based checkpoint selection with a randomly chosen intermediate checkpoint. The other reverses the order, training on the final teacher first and the intermediate checkpoint second. If the curriculum story is right, both should lose.
For published reference points, the strongest severity result on PhysioNet foot sensors is 88.67% overall 4-class accuracy, from the two-model pipeline in HCT: Hybrid Convnet-Transformer for Parkinson's disease detection and severity prediction from gait. Separately, the parallel 1D convolutional network reports 85.3% accuracy across 5 UPDRS severity classes under subject-level 10-fold cross-validation. We will reproduce under those protocols for comparability. For CARE-PD, no reported result found on Lacuna for UPDRS-gait score prediction, so our own reproduced cross-entropy and EEKD numbers are the bar.
Beating the baselines means two things. First, a gain in quadratic-weighted agreement and macro-F1 over the best of EEKD and the early-stopped one-shot teacher, consistent across held-out sites by a paired test over sites. Second, no loss on the rare severe class.
Why not just…?
Why not just early-stop the teacher and distill from that? This is the most serious threat. An early-stopped teacher may already avoid fitting rater noise, which would make the second stage pointless. The one-shot-from-early-stopped and one-shot-from-intermediate arms test this directly. The 2-shot schedule must beat both. Otherwise we conclude the benefit is "a less overfit teacher," not a curriculum, and we report that.
Why not just add auxiliary heads that predict speed, variability, and cadence? If the useful simple features are known, supervising them directly is cheaper than distilling them. On CARE-PD this could match our method, so it is run as a baseline with the same handcrafted targets our probes use. The method earns its place only if it matches that baseline on CARE-PD and beats it on PhysioNet, where the right sub-dimensions are not known in advance. If auxiliary heads win on both, the honest conclusion is that clinicians' sub-dimensions should be supervised explicitly.
Why not just smooth the labels? Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy? argues that distillation transfers sample-specific inter-class structure that uniform smoothing cannot. For a borderline patient, the model should spread probability between 1 and 2. For a clear case, it should not. The ordinal label-smoothing arm and the per-patient calibration analysis below test whether that difference shows up in clinical data.
Why not just use classical models on gait features? CARE-PD reports that deep motion encoders consistently beat handcrafted gait metrics across datasets. Even so, a gradient-boosted model on the probe features is included as a cheap floor.
Proposed Experiments and Decisive Checks
Primary metrics are quadratic-weighted agreement with the clinician score, macro-F1, mean absolute ordinal error, and recall on score 3. All are computed under leave-one-site-out evaluation on CARE-PD and subject-level 10-fold cross-validation on PhysioNet.
The first decisive check is a sample-efficiency curve. We subsample the training subjects to 10%, 25%, 50%, and 100%. The curriculum account predicts that the gap between the progressive student and the one-shot students widens as data shrinks. A constant or vanishing gap at low data would falsify the mechanism even if the full-data numbers look good.
The second check is mechanistic. Probe accuracy for simple kinematic quantities should peak at an intermediate checkpoint and decline afterward. The progressive student's own features should retain that decodability better than one-shot students do. If no intermediate peak appears, the selection rule has nothing to select, and the random-checkpoint ablation should then perform equally well.
The third check concerns subjectivity. On TULIP, we compare each model's predictive entropy per item with the observed disagreement among the three neurologists. We also measure whether model-versus-rater agreement falls inside the rater-versus-rater envelope. With 15 subjects this is descriptive rather than confirmatory. The prediction is that final-checkpoint teachers are confidently wrong on high-disagreement items, and progressive students are not.
Last, we report parameter count and inference cost for the small student. The point is to show the accuracy gains survive compression, not to claim compression as the contribution.
References
- Progressive distillation induces an implicit curriculum
- Computer vision uncovers three fundamental dimensions of levodopa-responsive motor improvement in Parkinson’s disease
- Deep 1D-Convnet for accurate Parkinson disease detection and severity prediction from gait
- Transformers for 1D signals in Parkinson's disease detection from gait
- CARE-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson's Disease Gait Assessment
- Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
- TULIP: Multi-Camera 3D Precision Assessment of Parkinson's Disease
- Towards Automated and Marker-Less Parkinson Disease Assessment: Predicting UPDRS Scores Using Sit-Stand Videos
- Learn From the Past: Experience Ensemble Knowledge Distillation
- Improved Knowledge Distillation via Teacher Assistant
- HCT: Hybrid Convnet-Transformer for Parkinson's Disease Detection and Severity Prediction from Gait
- Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?