Grading the Certainty of Video-Derived Gait Biomarkers

Phone-camera pose estimation now measures a patient's walking well enough to report numbers such as "hip range of motion is 53% lower after spinal cord injury." These numbers are presented with the same confidence whether the camera tracked the patient well or badly. The difficulty is that tracking error is not random noise. It grows sharply with pathology: joint position error rises from about 25 mm in healthy walkers to about 91 mm in spinal cord injury patients. Because the error differs between the groups being compared, it can bias the very difference the biomarker is meant to measure. Clinical epidemiology has a standard way to say how far a number should be trusted: GRADE, a checklist that downgrades a body of evidence for bias, inconsistency, indirectness, imprecision, and publication bias. Nobody has worked out what those downgrades mean when the "study" is a pose-estimation pipeline.

We propose a certainty grade for video-derived kinematic biomarkers. It makes the choices inside the measurement pipeline visible and counts them as sources of uncertainty. Those choices include which pose estimator is used, which coordinate frame, and how, or whether, the skeleton is rescaled to a standard body size. We then test whether the grade tells the truth. On a cohort where every video has simultaneous optical motion-capture ground truth, a biomarker graded "high certainty" should agree with the motion-capture value far more often than one graded "low." The two-sentence version is this. The spread of a gait biomarker across reasonable normalization and estimator choices, plus a calibrated estimate of how tracking error differs between patients and controls, becomes a GRADE-style certainty rating. We check that rating against motion capture.

Core Claim

Anatomical scale normalization is the family of transforms that remove camera position, body orientation, and body size from skeleton coordinates. In clinical measurement it plays two roles, and these roles have not been separated.

Its first role is a measurement choice with consequences for effect sizes. Joint angles do not change under uniform rescaling. Step length, stride width, and foot clearance do change. For those distance-type measures, choosing between absolute metres, leg-length-normalized units, or the uniform-bone-length skeletons used in gait recognition changes the size of the patient–control difference. Normalization choices also interact with pathology. Centering the skeleton on the pelvis moves pelvic tracking error into every other joint, and pelvic tracking fails most often in impaired walkers. The SCAI-Gait pipeline in 3D pose estimation for scalable remote gait kinematics assessment deliberately stays in the camera frame for exactly this reason.

Its second role is as a lens for grading. Once the variation across defensible pipelines is exposed, it maps onto GRADE's categories. Spread across equally defensible pipelines is a form of inconsistency. Differential tracking error between groups is a risk of bias. Mismatch between a normalized unit and the absolute quantity a clinical threshold uses is indirectness.

Our claim is that a grade built from these three pipeline-level quantities, together with ordinary sampling imprecision, predicts agreement with motion capture substantially better than the current practice of reporting a point estimate with a bootstrap interval over subjects.

Relation to Prior Work

The motivating work is SCAI-Gait 3D pose estimation for scalable remote gait kinematics assessment. It benchmarks VideoPose3D, RTMPose3D, BlazePose, and MotionBERT against Qualisys motion capture on 225 spinal cord injury patients. It reports range-of-motion deficits and a 96.5% classifier. It documents the fourfold error increase in patients but does not propagate that error into any statement of certainty. We reuse its data and its camera-frame design as one pipeline among several. Our addition is the certainty layer and its validation.

The normalization methods we vary come from gait and action recognition, where they were designed to increase invariance, not to preserve clinical meaning:

Two findings from this literature constrain our design. Fu et al. show that discarding stature helps cross-domain transfer but removes a real signal within a domain. This is the same tension clinically, where thresholds are often absolute. An et al. show that downstream results depend on which upstream detector is used. Both findings argue that the estimator and the normalization should be treated as uncertain choices rather than fixed preprocessing.

On the grading side, we borrow GRADE's five domains and four certainty tiers as applied in Liu et al. and Pan et al.. We also borrow the Evidence-to-Decision separation of certainty from recommendation strength described in Loo et al. and Ramaswamy et al.. Those works grade published trials by hand. We compute domain judgments from measurable pipeline quantities.

Can Large Language Models Match the Conclusions of Systematic Reviews? serves as a warning about automated grading. On text, language models take flawed evidence at face value; this is its "skepticism gap." Our design therefore keeps grading in explicit, auditable quantities rather than in a model's judgment.

Research Program

The unit of analysis is a biomarker contrast. Examples are the patient-minus-control difference in peak knee flexion during swing, or in leg-length-normalized step length, estimated from video. For each contrast we build a multiverse. This is the full grid of defensible pipelines, crossing four pose estimators with a small set of coordinate treatments: camera frame, pelvis-centered, Kabsch-aligned torso frame, neck-to-hip scaled, uniform bone length, and optional autoencoder denoising. Each pipeline gives an effect estimate $\hat\delta_p$ with a subject-bootstrap standard error.

We summarize uncertainty with an interval half-width:

$$ h = z\sqrt{s_{\text{samp}}^2 + s_{\text{pipe}}^2} + |\hat b| $$

The terms are defined as follows:

  • $s_{\text{samp}}$ is the typical sampling standard error across pipelines, which corresponds to GRADE imprecision.
  • $s_{\text{pipe}}$ is the standard deviation of $\hat\delta_p$ across pipelines, which corresponds to inconsistency.
  • $\hat b$ is the estimated differential measurement bias, which corresponds to risk of bias.
  • $z$ is the usual normal quantile.

The bias term is learned on a calibration split of subjects who have motion capture. We regress each pipeline's per-subject biomarker error on quantities available without motion capture: detector keypoint confidence, temporal jitter, bone-length variability across frames, and a severity proxy. We then apply that model to held-out subjects. Its prediction of the group-mean error difference is $\hat b$.

Indirectness is flagged when the biomarker's unit does not match the unit of the clinical threshold it is compared against. For example, the unit may be normalized when the threshold is in metres. When monocular video cannot recover absolute scale, the flag is structural, and the grade drops one tier.

Tiers are assigned GRADE-style. A contrast starts at High. It loses one tier per domain whose quantity exceeds a threshold set relative to a pre-registered minimal clinically important difference. Those thresholds are fixed on the calibration split only.

Finally, following Evidence-to-Decision logic, we pair each grade with an illustrative intervention of low or high burden. The purpose is to show where a moderate-certainty biomarker could still justify a low-burden home exercise referral but not a surgical referral. This last step is a demonstration, not a validated clinical claim.

Baselines

The naive reference point is what SCAI-Gait-style reporting does now. It takes the single best pipeline, reports its point estimate, and attaches a subject-bootstrap 95% interval. Nothing else counts as uncertainty. We run it on exactly the same held-out subjects with the same estimator outputs.

The strongest existing competitor for measurement accuracy is the SCAI-Gait pipeline itself. It uses VideoPose3D in the camera frame, with a reported PA-MPJPE of 90.7 mm in spinal cord injury and 24.8 mm in healthy subjects, plus reported range-of-motion reductions of 53.14% (left hip), 51.97% (right knee), and 56.28% (right hip), all from 3D pose estimation for scalable remote gait kinematics assessment. For the quantity we actually target, the coverage of the true effect by video-derived intervals, or the calibration of a certainty grade against motion capture, there is no reported result found on Lacuna. We therefore reproduce the baseline's intervals ourselves.

A second strong competitor is human GRADE judgment. Two methodologists, blind to motion capture, see each contrast's point estimate, interval, and per-group tracking-error summaries, and assign a tier by standard GRADE guidance. They receive the same information our method uses, minus the multiverse.

Three ablations isolate our ingredients:

  • multiverse spread only, which drops $\hat b$;
  • bias term only, which uses a single pipeline;
  • normalization held fixed while only estimators are varied, which tests whether normalization choices contribute beyond detector choice.

All methods share the same calibration/test subject split and the same pool of estimator runs. Thresholds for every method are tuned on the calibration split under the same search budget.

Beating the baselines means two things on held-out subjects. First, interval coverage of the motion-capture effect must be close to nominal, where the naive baseline undercovers. Second, the tier ordering must be monotone in agreement: the fraction of contrasts within one minimal clinically important difference of motion capture should strictly decrease from High to Very Low. We require the High–Low gap to be larger than the corresponding gap under human grading, with bootstrap confidence intervals over contrasts excluding zero.

Why not just…?

Why not just report a bootstrap confidence interval, which is already GRADE's imprecision domain? Resampling subjects captures sampling noise but cannot detect error that shifts one group more than the other. The documented 25 mm versus 91 mm tracking gap is exactly that kind of error. Our first decisive check compares the coverage of the naive interval with the coverage of our interval. If the naive interval already covers at nominal rates, the bias term is unnecessary, and we will say so.

Why not just run several pose estimators and report their disagreement? This is the most serious threat, because it is our "multiverse spread only" ablation. Estimators trained on similar healthy data may fail in the same way on impaired gait. If so, they agree with each other while all being wrong, which is why we also learn $\hat b$ against motion capture. But if the spread-only ablation matches the full method on coverage and tier calibration, the right conclusion is a simpler recommendation: report the multiverse spread and skip the learned bias model. We would publish that result.

Why not just use the most accurate estimator and skip normalization debates? An et al. show that downstream results change with the detector, and no estimator in SCAI-Gait is error-free on patients. More importantly, the choice between absolute and normalized distance units changes what quantity is being estimated, not just how precisely. The fixed-normalization ablation tests whether normalization adds variance beyond the choice of detector. If it does not, the normalization axis can be dropped for angle biomarkers and kept only for distance biomarkers.

Why not just have a language model read the pipeline outputs and assign a grade? MedEvidence reports a 60–62% accuracy ceiling for the best models versus over 75% for human experts. It also reports roughly 41% accuracy when sources conflict, and pipeline disagreement is exactly the conflicting-evidence case. We therefore do not expect a model grader to beat an explicit rule. We include a text-only model grader, given the same summaries as the human raters, only as a secondary reference.

Proposed Experiments

The primary experiment uses the SCAI-Gait Dataset (225 patients, more than 45,000 frames, Qualisys ground truth), for which access will be requested from its authors. It is paired with a healthy reference that is also motion-captured. For that reference we prefer healthy SCAI-Gait participants if they are available, and otherwise H3.6M and HumanEva-I. The fallback is weaker because the capture setting differs from the patient group, which could itself introduce bias. We define roughly forty contrasts, spanning hip and knee angles by gait phase and side plus distance measures in both absolute and leg-normalized units. We split subjects 50/50 into calibration and test sets, repeated over five seeds.

The first decisive check is interval coverage on test subjects: naive versus full method versus each ablation. The second is tier calibration. We measure agreement with motion capture within a pre-registered minimal clinically important difference for each tier, and compare the High–Low gap against human raters.

A secondary experiment addresses the heterogeneity problem that pushes observational meta-analyses to low certainty, the $I^2 > 90\%$ regime in Huang et al.. We simulate small "studies" by drawing subsets of 20–40 patients, each processed with a different randomly chosen pipeline. We then meta-analyze them by random effects and compare $I^2$ and pooled-effect error against motion capture in two conditions: with pipelines left heterogeneous, and with every study harmonized to one shared normalization, using segment resampling in the style of Yu et al. where joint topologies differ. If harmonization cuts $I^2$ without moving the pooled estimate away from motion capture, pipeline heterogeneity is a fixable cause of low certainty in movement meta-analyses. If it cuts $I^2$ but increases error, harmonization hides disagreement rather than resolving it. That outcome would argue for grading pipeline spread explicitly, as proposed here, rather than normalizing it away.