Home rehabilitation software watches a patient exercise through a webcam or smartwatch and, after each repetition, says either "good" or "fix this." The models behind those verdicts are trained against one source of truth. That source is a single therapist's recorded demonstration, a single therapist's scores, or a single scalar score per recording. Physical therapists differ in how much trunk lean or shortfall in range of motion they will accept. Work on medical imaging shows that a network trained on one expert does more than inherit that expert's habits: it exaggerates them. We propose to measure how often current exercise-feedback models tell a patient to correct a repetition that most therapists would have accepted. We then build a feedback model that predicts what fraction of a therapist panel would accept each repetition. Its policy is to issue a correction only when that panel would broadly agree, and to give neutral encouragement when therapists themselves would split.
The bet is that a large share of the negative feedback patients receive at home is clinically contested rather than clinically wrong. If so, removing it should cost little in movement quality, while sparing patients repeated rebukes they cannot act on. Those rebukes are a plausible driver of the drop in adherence once supervision ends. The project has three parts. The first is a new multi-therapist re-annotation of public rehabilitation recordings. The second is a test of whether single-rater bias amplification shows up in movement assessment over time, not only in static images. The third is a pilot study of whether disagreement-aware feedback changes how patients behave.
Core Claim
The main ingredient is not a fancier architecture. It is a change in what the model is trained to predict. Instead of one score or a majority label, the model predicts $p(x)$, the expected fraction of a therapist panel that would accept repetition $x$. A separate head predicts which correction reasons (for example trunk compensation, short range, or rushed tempo) panel members would cite. The feedback rule is then
$$\text{correct}(x) \iff p(x) < \tau_{\text{low}}, \qquad \text{praise}(x) \iff p(x) > \tau_{\text{high}}, \qquad \text{neutral otherwise},$$
where $\tau_{\text{low}}$ and $\tau_{\text{high}}$ are thresholds chosen on validation data so that "correct" is issued only when a clear majority of held-out therapists would also correct.
We make two claims, each testable. First, models trained on a single therapist produce systematically more contested corrections than the therapist would. This would be the temporal analogue of the amplification effect reported for lesion segmentation. Second, the panel model matches an unseen therapist's judgments better than any single-rater model, and at the same missed-correction rate it issues far fewer contested corrections.
Grounding in Prior Work
Three lines of evidence motivate the proposal.
The Impact of an Inter-rater Bias on Neural Network Training trained identical 3D U-Nets on two raters' brain-lesion annotations. It showed that a classifier could tell the two raters apart more easily from the networks' outputs (hit rates 0.90–0.93) than from the original human annotations (0.87–0.917). In other words, the networks sharpened each rater's style.
Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment found that single-pathologist training gave unstable models (F1 interquartile range 0.038), that a three-rater consensus roughly halved that spread (0.016), and that adding raters beyond three gave only about +0.013 F1. That result constrains our annotation budget: a panel of about six therapists should be ample.
Discordance in diagnosis among eight expert dermatopathologists, who fully agreed on only 53.5% of lesions (Discordance, accuracy and reproducibility study of pathologists' diagnosis of melanoma and melanocytic tumors), undercuts the idea that a single "master" rater exists. All of this evidence comes from static images. None of it touches movement scoring, and none of it asks what happens downstream when the model's verdict is shown to a patient.
On the rehabilitation side, the systems we build on all anchor their output to one reference. Home-Based Physical Therapy with an Interactive Computer Vision System (ExerciseCheck) aligns patient joint trajectories to a single recorded therapist using dynamic time warping, an algorithm that stretches two time series so they line up before comparing them. That paper also reports roughly 10 cm average 3D joint error from webcam pose estimation. PhysiQ: Off-site Quality Assessment of Exercise in Physical Therapy scores smartwatch-recorded shoulder exercises against a single Siamese anchor and reports 89.67% quality classification accuracy. VI-Net: View-Invariant Quality of Human Movement Assessment regresses one scalar quality score per recording on KIMORE and QMAR. Automated Movement Assessment in Stroke Rehabilitation (SARAH) uses therapist ground truth without modeling therapist variability, and leaves real-time feedback as future work. Our proposal reuses these systems' encoders and datasets. What we change is the target, and what we evaluate is the feedback the patient would actually see.
Two methods are closest to ours. Transformer-based Annotation Bias-aware Medical Image Segmentation (TAB) learns a separate preference embedding per annotator on top of a shared image backbone. UDF-GMA: Uncertainty Disentanglement and Fusion for General Movement Assessment separates model ignorance (epistemic uncertainty, estimated with Monte Carlo dropout) from data noise (aleatoric uncertainty) for infant movement screening. It shows that discarding the 60% most uncertain cases lifts sensitivity on the remainder to nearly 99%. We differ from TAB in modality and in purpose: our aim is a calibrated estimate of panel agreement that drives what the patient hears, not an imitation of each annotator. We differ from UDF-GMA because its uncertainty contains no information about whether therapists would disagree. Our panel labels let us test directly whether that kind of uncertainty tracks real clinical disagreement.
Research Program
The first stage is data. No multi-therapist movement dataset appears in our search, so we will build one. We will re-annotate repetitions from KIMORE (78 subjects, lower-back rehabilitation exercises) and from QMAR (38 subjects, walking and sit-to-stand across six camera views, with simulated abnormalities graded by severity). We will add the PhysiQ shoulder recordings if they can be obtained. Six licensed physical therapists from at least two clinics will each rate every repetition as accept or needs-correction, tag the reasons, and rate their own confidence. A random 15% of repetitions will be re-presented to the same rater weeks later, so we can measure each therapist's consistency with themselves. Without that, inconsistency within one rater would be mistaken for disagreement between raters. Ratings will be made from the same video the model sees, so disagreement is not confounded by therapists having better viewing conditions than the model.
The second stage is diagnosis. We will train one model per therapist with a fixed encoder and test three things. First, whether a classifier separates therapists more easily from those models' outputs than from the therapists' own labels, which repeats the amplification test on time-series data. Second, how each single-rater model's correction rate compares with its own rater's correction rate. Third, how many of each model's corrections fall on repetitions that a majority of the other five therapists accepted.
The third stage is the panel model. We will use a graph-convolution encoder over pose sequences (and VI-Net's view-invariant trajectory module for QMAR), with an acceptance-fraction head trained by cross-entropy against soft panel labels and a multi-label reason head. Per-annotator embeddings in the style of TAB will be an optional extension. They would let a prescribing therapist choose their own tolerance, but they are not the core claim.
The fourth stage is a pilot engagement study, described under the experiments below.
Baselines
The naive reference point is the existing single-score pipeline: VI-Net trained on KIMORE's original clinical scores, thresholded to accept or correct. VI-Net reports a Spearman rank correlation of 0.66 on KIMORE against a prior 3D temporal baseline at 0.62 (VI-Net). These are the only reported numbers on our target data. We found no reported result on Lacuna for any model evaluated against multiple therapists' repetition-level judgments, so every panel-level number in this project will be new. We will retrain VI-Net on our splits rather than copy its figure, and check that we reproduce roughly 0.66 before comparing.
The single-rater baselines are six models, one per therapist. We also train a seventh on the most experienced therapist, which is the "master rater" alternative. This tests the inter-rater bias paper's observation that the more experienced rater gave less biased networks.
The template baseline follows ExerciseCheck: dynamic time warping against a single therapist's demonstration, with the distance threshold tuned on validation data. The kinematic-rule baseline thresholds joint angles and trunk-lean angles computed from the same pose estimates.
The strongest competitor is a mean-consensus model. It uses the same encoder, trained on the panel's majority vote or mean score, with a tuned decision threshold. The mitotic-figure study suggests consensus training captures most of the stability benefit, which is why this is the hardest baseline to beat.
The uncertainty baseline is that consensus model plus Monte Carlo dropout, set to abstain or give neutral feedback when uncertain, as UDF-GMA does.
The key ablation isolates our ingredient. It uses an identical encoder, data, and three-way feedback rule, but replaces soft panel labels with hard majority labels and uses predicted confidence in place of $p(x)$.
For fairness, every learned baseline shares the encoder, subject-disjoint splits, training budget, and a fixed number of hyperparameter trials, and all thresholds are set on the same validation subjects. To count as beating the baselines, the panel model must do three things on held-out subjects, judged by a held-out therapist. It must reduce the contested-correction rate (corrections on repetitions the held-out therapist and a majority of the others accepted) at a matched missed-correction rate, with subject-bootstrap confidence intervals that exclude zero. Its negative log-likelihood for the held-out therapist's label must be no worse than any single-rater model's. And its predicted $p(x)$ must be calibrated against the observed panel fraction.
Why not just…?
Why not just average the therapists and train on the consensus? This is the most serious threat, and we take it seriously rather than argue around it. Consensus training may well match our model on accuracy against held-out raters. The mitotic-figure study found that three-rater consensus closed most of the stability gap, and that model performance hit a ceiling of about 0.74–0.76 F1 regardless of label quality. Where consensus should fall short is the feedback decision. A majority label cannot tell a repetition that therapists accept 4-to-2 from one they accept 6-to-0, so it cannot route contested repetitions to neutral feedback. The hard-label ablation tests exactly this. If thresholding a consensus model's confidence produces the same operating curve as $p(x)$, then our contribution shrinks to the dataset and the diagnosis, and we will report it that way.
Why not just use objective kinematic thresholds and skip therapists altogether? Joint angles sound objective. But ExerciseCheck measured roughly 10 cm of 3D joint error from webcam pose estimation, and PhysiQ notes that a single wrist sensor cannot see trunk compensation. Thresholds on noisy angles therefore encode measurement error rather than clinical judgment. The kinematic-rule baseline settles this empirically. We will also regress panel acceptance on kinematic features to report how much of the therapists' judgment those features explain at all.
Why not just use one excellent senior therapist? The melanoma panel's 53.5% full agreement among top experts suggests no single rater is a reliable gold standard, and the amplification result suggests a model will exaggerate whichever rater it learns from. The master-rater baseline is evaluated against the other five therapists under leave-one-rater-out testing. If it matches the panel model, that is a cheaper recipe, and we will say so.
Why not just use Monte Carlo dropout uncertainty and abstain? Abstaining on 60% of repetitions, as UDF-GMA's filtering did, would leave a home patient without feedback most of the time. More fundamentally, dropout measures model ignorance, not clinical disagreement. We will directly compute the rank correlation between dropout variance and the observed spread of panel judgments. If that correlation is high, the uncertainty baseline is a cheap substitute and our labels mainly serve to validate it.
Proposed Experiments
The first decisive check is amplification. Do single-therapist movement models become more distinguishable from one another than the therapists' own labels, and do they correct more often than their rater does? A null result here would itself be informative. It would mean movement scoring is less vulnerable than segmentation, which weakens the motivation for the rest of the project.
The second check is generalization to an unseen therapist, run as six-fold leave-one-rater-out on subject-disjoint splits. It compares the panel model with every baseline on contested-correction rate, missed-correction rate, held-out log-likelihood, and calibration of $p(x)$. We will also plot held-out performance as the number of training raters grows from one to five, which tests whether the three-rater saturation seen in histology also holds for movement.
The third check is the engagement pilot. About 30 adults with lower-back pain will do a home program in a two-period crossover, receiving single-rater feedback in one period and panel-aware feedback in the other, with the order randomized. The patient-facing measures are sessions completed, repetitions performed relative to the prescription, and a short frustration and perceived-competence questionnaire after each session. The safety measure is movement quality, judged by therapists blind to condition from recorded repetitions.
The pilot succeeds if adherence is non-inferior or better under panel-aware feedback and blinded quality is not worse. A drop in quality would show that contested corrections were doing useful work. We will pre-register the primary outcome as sessions completed and treat everything else as exploratory, since a sample of this size can detect only large effects.
References
- The Impact of an Inter-rater Bias on Neural Network Training
- Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
- Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors
- Home-Based Physical Therapy with an Interactive Computer Vision System
- PhysiQ: Off-site Quality Assessment of Exercise in Physical Therapy
- VI-Net - View-Invariant Quality of Human Movement Assessment
- Automated Movement Assessment in Stroke Rehabilitation
- Transformer-Based Annotation Bias-Aware Medical Image Segmentation
- UDF-GMA: Uncertainty Disentanglement and Fusion for General Movement Assessment