Models that score Parkinson's disease severity from walking videos or motion capture look accurate inside one clinic and then fail at a new one. The field's current response is to train bigger motion encoders or to fine-tune on a few labeled walks from the new clinic. Neither tells us why a gait measure breaks when it moves between sites. There are two quite different ways it can break. In the first, a clinic simply records everyone's stride as a bit longer, because of camera setup or track length; this is an offset, and re-centering within each site fixes it. In the second, the relationship between the measure and clinical severity differs from site to site; no normalization repairs that. Random-effects meta-analysis is the standard statistical tool for pooling an effect across studies whose true effects differ. It estimates the average effect and the between-study spread of true effects, written τ². It can tell these two failure modes apart, one feature at a time, using only the training sites.
We propose to treat each clinical cohort in a multi-site gait archive as a separate retrospective study. For every candidate kinematic feature we estimate its severity association within each cohort, adjusted for medication state, and pool those associations with a random-effects model. The resulting τ², and the prediction interval it implies for a not-yet-seen site, serve as a transportability score. The claim to test is concrete. Slope heterogeneity measured on source sites alone should predict which features, and which deep models, hold up on a held-out clinic. It should also tell us whether handcrafted gait features really fail across sites, or only fail when they are pooled naively.
Core Claim
Cross-site failure of gait severity models has two parts. One is removable site offsets. The other is irreducible heterogeneity in the feature–severity slope. The second is estimable before deployment. For site $i$, subject $j$ with ordinal severity $y_{ij}$ (UPDRS-gait 0–3), feature value $x_{ij}$ standardized within site, and medication state $m_{ij}$ (ON/OFF), we fit a per-site ordinal logistic model with site-specific thresholds and slope $\beta_i$. We then assume
$$\hat\beta_i \sim \mathcal{N}(\mu, \tau^2 + s_i^2),$$
where $\mu$ is the average log-odds of higher severity per within-site standard deviation of the feature, $\tau^2$ is the between-site variance of true slopes, and $s_i^2$ is the sampling variance of site $i$'s estimate. The quantity we care about is the random-effects prediction interval for a new site's slope, $\mu \pm t_{K-2}\sqrt{\tau^2 + \mathrm{SE}(\hat\mu)^2}$, where $K$ is the number of sites and $t_{K-2}$ is the Student-t quantile. A feature is "transportable" when this interval excludes zero and stays narrow. The prediction is that, computed on $K-1$ sites, this interval ranks features, and whole models, by their macro-F1 on the held-out $K$-th site.
This is new relative to the closest work. CARE-PD assembles nine cohorts from eight centers into a common 3D body-mesh format. It evaluates leave-one-dataset-out (LODO) generalization and target-site fine-tuning (MIDA), and reports that handcrafted features transfer worse than motion encoders. It never partitions within- from between-site variance or asks whether the handcrafted deficit is an offset artifact. We reuse its data and protocol but change the question from "which model generalizes" to "which associations are stable, and can we tell in advance."
Relation to Prior Work
Some prior work motivates the gap by pooling cohorts without modeling them. Naimi et al., Nguyen et al. and El Maachi et al. concatenate the three PhysioNet sub-studies (Hausdorff, Yogev, Toledo) and report 10-fold accuracy. That mixes sites across folds, so their numbers say nothing about transfer. Shi et al. go the other way: they analyze two accelerometry cohorts separately because hip/wrist placement and follow-up differ. They never ask which associations the cohorts share.
Other work constrains the design. Evers et al. find that cadence falls apart in free-living data while simple 0.5–10 Hz spectral power tracks medication response. That is a warning that "handcrafted" is not one category, and that stability is feature-specific. Our per-feature audit is designed to expose exactly this. Giacomini et al. show that standard empirical-Bayes shrinkage pulls genuine outliers toward the group mean. For us, this means shrunken site-specific slopes could hide a truly deviant clinic. We therefore report unshrunk per-site slopes alongside pooled ones and do not use shrinkage to declare sites homogeneous.
The method itself is supplied by standard random-effects pooling, as used across the observational meta-analyses in the file, for example the 29-cohort synthesis of Luo et al.. Medication and FoG adjustment borrows the retrospective-cohort logic of controlling confounding by indication within each cohort before comparing across them. Simchoni and Rosset's LMMNN adds random effects inside a neural network. We use it as a competitor rather than as our method, for reasons given below.
Research Program
The primary testbed is CARE-PD: 362 subjects, 8,477 walking segments, UPDRS-gait labels, ON/OFF status and FoG flags, all in a common SMPL mesh representation. We extract a fixed bank of roughly forty interpretable features from the meshes. These include spatiotemporal parameters (cadence, step length, double-support fraction, step-time variability), smoothness measures (normalized jerk of pelvis and ankles), spectral power bands after Evers et al., arm-swing amplitude and asymmetry, and trunk flexion. Each feature is computed per subject by averaging segments. This makes the subject the unit of analysis, which avoids pseudo-replication across walking segments.
Each feature gets three analyses. First, we compare pooled-standardized against within-site-standardized versions. This measures how much cross-site failure is offset alone. Second, we run the per-site adjusted ordinal slope and random-effects pooling described above, estimating τ² by restricted maximum likelihood with Hartung–Knapp-style intervals, because $K \le 9$ makes τ² itself uncertain. Third, we run a meta-regression of site slopes on recorded site descriptors, namely capture modality (video versus optical motion capture), frame rate and walkway length. This asks which technical factors explain τ².
Deep models get the same treatment. For MotionAGFormer and MoMask trained under LODO, we treat each model's continuous predicted severity on the held-out site as a single "feature." We then compute its per-site calibration slope over the rotation of held-out sites. This puts learned and handcrafted biomarkers on a common heterogeneity scale.
The second testbed is PhysioNet's three force-plate sub-studies. With $K=3$, it probes the regime where τ² is barely identifiable. Here the note makes a weaker claim: the audit should at least flag wide intervals honestly rather than certify stability. Its severity labels (Hoehn–Yahr stages) differ from CARE-PD's. That is a deliberate check that the audit is not tied to one outcome scale.
Baselines
The naive reference point is direct concatenation: a single ordinal classifier on pooled-standardized handcrafted features, trained on all source sites with site ignored, then evaluated on the held-out site. This is what the PhysioNet papers effectively do, and it is the setting in which CARE-PD reports handcrafted features lagging.
The strongest existing method is the motion encoders under CARE-PD's LODO protocol. We use MotionAGFormer and MoMask with the released configurations and the same subject splits, and give their tuning budget to all other methods. We found no reported result on Lacuna for UPDRS-gait macro-F1 or accuracy under LODO or MIDA in CARE-PD. The only CARE-PD number in the file is a reconstruction error, MPJPE from 60.7 mm to 7.5 mm for MotionAGFormer, which is not a severity metric. We therefore reproduce LODO ourselves. For context on the secondary benchmark, published PhysioNet figures are 85.3% severity accuracy for El Maachi et al. and 87.89% overall accuracy for Naimi et al.. Both come from 10-fold splits that mix sub-studies, so we expect them to drop under leave-one-study-out, and we re-run the Naimi architecture under that protocol.
The target-site upper reference is MIDA fine-tuning with a small labeled fraction of the target clinic. We do not expect to beat it. It uses labels our setting forbids, and we report it only to bound the gap.
The key ablation isolates the random-effects ingredient. We build a "transportable subset" model from features selected on source sites by prediction-interval criteria. We compare it with the same model using features selected by pooled-significance (fixed-effect) criteria, and with features selected by per-feature source-site LODO performance. Beating the baselines means two things. The τ²-selected, within-site-standardized handcrafted model must match or exceed the motion encoders' mean LODO macro-F1, within the across-site standard error. And τ²-based selection must beat fixed-effect selection on held-out sites.
Why not just…?
Why not just standardize features within each site and stop there? This is the most serious threat, and it is built in as a condition rather than argued away. If within-site standardization alone closes the handcrafted-versus-deep LODO gap, the answer to the core question is "heterogeneity is mostly offsets." That would be a useful and publishable negative for the τ² ranking. The ranking claim survives only if, after standardization, features still differ substantially in estimated τ², and those differences predict held-out performance. One caveat: within-site standardization also changes the label distribution each feature sees, because sites differ in severity mix. We therefore check it against a version that standardizes only on each site's score-0/1 subjects.
Why not just put random site effects into the network, as LMMNN does? LMMNN improves prediction when test data come from groups seen in training. At a genuinely new clinic, the random intercept is unknown and defaults to zero, so the prediction reduces to the fixed-effect network. We will run LMMNN with a site random intercept on top of the motion-encoder embeddings under LODO to confirm this empirically. We expect no LODO gain over the plain encoder, and a gain only when some target labels are available. That second case is MIDA's setting, not ours.
Why not just measure each feature's LODO performance directly and pick the best? This is the empirical alternative to the meta-analytic score, and it is also a real competitor. With nine sites, per-feature LODO estimates are noisy. They also mix offset failure with slope failure, and they give no uncertainty for a tenth site. The nested experiment below settles it. Both selection rules are computed on eight sites and scored on a ninth never used for selection. If raw LODO selection does as well, we report that the meta-analytic apparatus adds interpretability but not accuracy.
Why not just adjust for age and disease duration with propensity scores? CARE-PD does not uniformly release those covariates, so full propensity balancing is not available across cohorts. We adjust for what is recorded, ON/OFF state and FoG, within each site. We treat residual confounding as a named limitation. A sensitivity analysis on the subset of cohorts with extra covariates tests whether slopes move.
Proposed Experiments and Decisive Checks
The main experiment is nested leave-one-site-out on CARE-PD. For each held-out site, all τ² estimates, feature selection and model fitting use only the remaining sites. We then record held-out macro-F1, balanced accuracy and ordinal mean absolute error. The first decisive check is rank agreement. Across features, the Spearman correlation between source-site prediction-interval width (or the sign-excluding criterion) and held-out single-feature macro-F1 should be clearly positive and larger than the correlation for fixed-effect pooled significance. If it is near zero, the core claim fails.
The second check is the handcrafted-versus-deep comparison under matched standardization. It asks whether the deficit reported in CARE-PD persists once offsets are removed and heterogeneous features are excluded. The third is meta-regression: whether capture modality or frame rate explains a meaningful share of τ² for smoothness and jerk features. We expect these to depend on frame rate and to be most sensitive to video-derived meshes. The fourth, on PhysioNet with $K=3$, checks that the audit reports intervals wide enough to cover held-out slopes. A calibration failure there would indicate that the approach should not be trusted with few cohorts.
Finally, we apply the same audit to the motion encoders' predicted scores. If a deep model's calibration slope shows lower τ² than every handcrafted feature, that is direct evidence that learned representations are genuinely more transportable, and not merely better fit to the dominant sites.
References
- CARE-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson's Disease Gait Assessment
- 1D-convolutional transformer for Parkinson disease diagnosis from gait
- Transformers for 1D signals in Parkinson's disease detection from gait
- Deep 1D-Convnet for accurate Parkinson disease detection and severity prediction from gait
- Free-living wrist and hip accelerometry forecast cognitive decline among older adults without dementia over 1- or 5-years in two distinct observational cohorts
- Real-Life Gait Performance as a Digital Biomarker for Motor Fluctuations: The Parkinson@Home Validation Study
- Individual Shrinkage for Random Effects
- Risk factors of epidural hematoma in patients undergoing spinal surgery: a meta-analysis of 29 cohort studies
- Integrating Random Effects in Deep Neural Networks