Phone-video gait analysis promises to bring lab-grade walking measurement to any clinic. A pose estimator, which is a neural network that locates body joints in each video frame, turns ordinary footage into hip and knee angles. The catch is that these systems are validated where an optical motion-capture lab exists to supply ground truth. The clinics that most need cheap video analysis are the ones that have no lab and so have no way to check whether the system works on their cameras, their rooms and their patients. We propose to use something every clinic already produces as the check: the physical therapist's free-text note. When a note says "decreased R knee flexion in swing, circumducts, uses RW," it records an independent clinical measurement of the same gait. If we extract those statements reliably and compare them with what the video pipeline reports, the per-site rate of disagreement could estimate the pipeline's true error at a new site without motion capture. It could also tell us which pose estimator to deploy there.
Put briefly, the bet is this. Video errors come from the site's cameras and rooms. Note errors come from the site's documentation habits. These two sources of institutional bias are mostly unrelated, so agreement between the two channels carries information about each one that neither carries alone. The note tests whether a measure-the-measurer signal built from routine clinical text can replace a motion-capture lab when deciding whether accessible gait analysis has survived a move to a new institution.
Core Claim
At a held-out clinic, a corrected agreement score between gait deviations extracted from notes and gait deviations derived from video will do two things. It will predict the motion-capture-referenced error of each markerless pipeline. It will also rank pipelines in the same order the motion-capture reference would. Both should hold more accurately than signals computed from the video alone, such as keypoint confidence or bone-length consistency across strides. If the claim fails in a specific way, namely that video-only consistency matches the note-based score, then the text channel adds nothing and the idea should be dropped. The experiments are built to detect that outcome.
Why This Gap Is Real
The evidence that single-site validation misleads comes from other modalities. Pathology foundation models memorize their source institution, and UNI's patch accuracy falls from 0.89 to 0.44 when institutional correlations are inverted (Unveiling Institution-Specific Bias in Pathology Foundation Models). In gait video specifically, abnormality classifiers drop from 91–97% accuracy on front and back views to 56–66% on side views (Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset). That is exactly the kind of geometry that changes from clinic to clinic.
The strongest clinical markerless study benchmarks five 3D pose estimators on 225 spinal cord injury patients against Qualisys motion capture. VideoPose3D wins, with a Procrustes-aligned mean per-joint position error (PA-MPJPE, the average joint distance after optimally rotating and scaling the prediction onto the truth) of 90.7 mm, against 24.8 mm on healthy walkers (3D pose estimation for scalable remote gait kinematics assessment). That study motivates our work, but it was conducted at one center under one protocol. We do not know whether VideoPose3D is still the right choice anywhere else. The question of whether algorithm rankings survive institutional shift is precisely the open problem the comparative-evaluation literature leaves untouched.
On the text side, rehabilitation documentation is described as a "Wild West" of shorthand, and fewer than half of recommended neurologic and orthopedic assessments exist in the OMOP common data model (Challenges and opportunities of using common data models in rehabilitation). That paper is a perspective and runs no extraction benchmark. The closest empirical work, Precision Rehabilitation for Patients Post-Stroke, extracted exercises and AM-PAC mobility scores from single-site therapy notes with rules. It never linked them to measured kinematics.
The method we borrow comes from electrocardiography. There, cardiologist narrative over-reads were used to expose systematic errors in machine interpretation (Beyond Machine Interpretation: Learning from Expert Over-Reads Improves ECG Diagnosis). That work used the text to train a better model at one health system. We use clinician text differently, as an evaluation instrument across health systems, and in a modality where nobody has tried it.
Research Program
The first component is an extractor that turns therapy notes into a fixed schema of gait findings. The schema covers, for each joint and side, reduced range of motion and specific deviations such as circumduction, foot drop and Trendelenburg sign, along with assistive device use and coarse cadence or speed category. Each finding has three possible values: present, absent or unmentioned. The extractor follows the self-verification design of Self-Verification Improves Few-Shot Clinical Information Extraction. A language model extracts findings, checks for omissions, quotes the supporting text span, and prunes any claim it cannot ground. Grounding matters here because every disagreement has to be auditable by a clinician. The ontology check in mCODEGPT, which validates leaf nodes and reports a 1.1% hallucination rate on oncology entities, supplies the template for restricting outputs to schema terms.
The second component maps each pose pipeline's output to the same schema. A finding is flagged when a measured quantity crosses a threshold, for example swing-phase knee flexion more than a set margin below the age-matched normal. The thresholds are fixed once from public healthy data and are not tuned per site.
The third component is the estimator. For a binary finding, let $q_s$ be the note extractor's accuracy at site $s$ against chart review, and let $p_{s,m}$ be pipeline $m$'s accuracy against motion capture. If the two error sources are independent, the observed agreement $A_{s,m}$ satisfies
$$A_{s,m} = p_{s,m}\,q_s + (1-p_{s,m})(1-q_s), \qquad \hat p_{s,m} = \frac{A_{s,m} - (1-q_s)}{2q_s - 1}.$$
We estimate $q_s$ from roughly a hundred hand-annotated notes per site. That annotation is far cheaper than a motion-capture lab, and it absorbs each site's documentation culture. The resulting $\hat p_{s,m}$ is the label-free estimate of pipeline accuracy.
Independence is the assumption most likely to break. A therapist and a camera can both miss a subtle deviation. We therefore measure this correlation directly at sites that have motion capture, and we report how far the correction degrades as the correlation grows.
The data requirement is substantial and we state it plainly. No dataset in the literature pairs clinical gait video with therapy notes. The program therefore needs a new multi-site cohort of four or more outpatient rehabilitation sites with differing camera hardware and room layouts. At least two sites must have optical motion capture, so that leave-one-site-out evaluation has ground truth. Each patient contributes a phone video, a same-day therapist note, and, at the lab sites, synchronized motion capture plus a waist and shank wearable sensor (an inertial measurement unit, or IMU) recording.
Baselines
The naive reference point is the current practice of trusting the source site. Under this baseline, the pipeline chosen at the development center, VideoPose3D following 3D pose estimation for scalable remote gait kinematics assessment, is deployed everywhere, and its in-distribution cross-validated error is assumed to hold. The reported number to beat is PA-MPJPE of 90.7 mm on spinal cord injury gait, from that paper. Our method beats this baseline if, at held-out sites, the agreement-selected pipeline has lower motion-capture error than the fixed choice, or if $\hat p$ flags sites where the fixed choice's error exceeds its source-site value.
The strongest video-only competitor is an internal-consistency score computed from the pose output itself. It combines mean keypoint confidence, bone-length variance across strides (the anatomical prior used in the cited monocular work), and left-right timing consistency of heel-strike events. Event timing is known to be reliable from off-the-shelf trackers (Concurrent validity of human pose tracking in video). This baseline is the most dangerous because it costs nothing and needs no text. It runs on identical videos, and its weights are fit on the same development sites used to set our thresholds. Beating it means higher Spearman correlation with true per-site, per-pipeline error at held-out sites, and higher Kendall rank agreement with the motion-capture ranking.
The strongest sensor competitor is a wearable IMU worn during the same walk. Its joint angles and gait events serve as a portable reference against which video is scored. No reported result was found on Lacuna for IMU-referenced validation of markerless video at new clinics. We expect IMU to be more accurate than notes at measuring individual joints. Notes win only if they come close at zero added collection cost, and we will report the gap honestly.
On the extraction side, the baseline is a rule-based extractor of the kind built in Precision Rehabilitation for Patients Post-Stroke. Its abbreviation dictionary is written at one site and then applied unchanged at the others. The language-model extractor gets no more site-specific tuning than the rules do. No reported result was found on Lacuna for extracting gait deviations from rehabilitation notes. The nearest reported figure for the self-verification method is a MIMIC-IV ICD-10 F1 rising from 0.487 to 0.533 (Self-Verification Improves Few-Shot Clinical Information Extraction).
Two ablations isolate the key ingredient. The first uses raw agreement $A_{s,m}$ without the per-site note-accuracy correction. It tests whether modeling documentation-culture bias matters. The second fits a combined model on internal consistency plus note agreement. It tests whether the text adds information beyond what video consistency already supplies.
Why not just…?
Why not just collect a few motion-capture trials at each new site? Wherever this is possible, it is the better instrument. The target setting, though, is clinics with no lab, which is the whole point of accessible gait analysis. Optical capture is restricted to laboratory environments (Verification of Markerless Gait Analysis). Within the lab sites, we will still measure how many motion-capture trials are needed to match the ranking accuracy of the note-based score. That number tells a site with partial access which route to take.
Why not just use the pose estimator's own confidence and bone-length consistency? This is a real threat. The concern is that pathological gait can produce errors that are consistently wrong. A model trained on healthy walkers can smoothly underestimate the range of motion of a contracted knee, with stable bone lengths and high confidence. The fourfold error inflation from healthy to spinal cord injury gait in 3D pose estimation for scalable remote gait kinematics assessment suggests this happens, but it does not prove it. The combined-model ablation settles the question. If adding note agreement does not raise held-out correlation with true error, the text channel is unnecessary and we will say so.
Why not just use IMUs as the portable reference? Wearables add a device, a protocol and a calibration step to every visit, and that reintroduces site-specific procedure. Notes are already written for every visit. We run the IMU reference head to head. If it beats notes by a wide margin, the honest conclusion is that clinics should buy sensors. The note approach would then keep its role only where sensors are impractical.
Why not just validate with k-fold cross-validation and augmentation at one strong site? The pathology collapse from 0.89 to 0.44 (Unveiling Institution-Specific Bias) and the gait viewpoint collapse (Computer Vision for Clinical Gait Analysis) both show that in-distribution validation hides exactly the failures at issue. The naive baseline above measures how wrong source-site estimates turn out to be.
Proposed Experiments
A cheap pilot comes first, before any new data collection. On GAVD, we treat its clip sources as pseudo-sites and its clinician-assigned abnormality subclasses as a coarse stand-in for narrative findings. The check is whether agreement between pose-derived deviation flags and those labels drops on side-view clips, where accuracy is already known to collapse to 56–66%, and whether it orders the five pose estimators consistently. CASD gives a second, lab-controlled domain for the same check. If agreement cannot even detect a failure we already know exists, the main study is not worth running.
In the main study, every pose pipeline benchmarked in 3D pose estimation for scalable remote gait kinematics assessment, including RTMPose3D, BlazePose, MotionBERT and VideoPose3D, runs on all sites. Error is defined against motion capture at the lab sites, as PA-MPJPE and as per-finding accuracy $p_{s,m}$. Evaluation uses leave-one-site-out, and every threshold, consistency weight and extraction prompt is frozen before a held-out site is examined.
The first decisive check is whether $\hat p_{s,m}$ tracks true $p_{s,m}$ across held-out site and pipeline pairs better than internal consistency does. The second is whether the agreement-based ranking of pipelines at a held-out site matches the motion-capture ranking more often than chance and more often than the source-site ranking. The third is whether the measured correlation between note errors and video errors stays small enough for the correction to help, which we test by comparing raw against corrected agreement. As a side result, the study yields the first cross-site comparison of rule-based and self-verifying language-model extraction on rehabilitation notes, reported per site to expose how much documentation culture moves each method.
The idea fails cleanly if the note-based estimate adds no held-out predictive value over video-only consistency, or if note and video errors turn out to be strongly correlated on pathological gait.
References
- Unveiling Institution-Specific Bias in Pathology Foundation Models: Detriments, Causes, and Potential Solutions
- Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset
- 3D pose estimation for scalable remote gait kinematics assessment
- Challenges and opportunities of using common data models in rehabilitation
- Precision Rehabilitation for Patients Post-Stroke based on Electronic Health Records and Machine Learning
- Beyond Machine Interpretation: Learning from Expert Over-Reads Improves ECG Diagnosis
- Self-verification improves few-shot clinical information extraction
- Introducing mCODEGPT as a zero-shot information extraction from clinical free text data tool for cancer research
- Concurrent validity of human pose tracking in video for measuring gait parameters in older adults: a preliminary analysis with multiple trackers, viewing angles, and walking directions
- Verification of Markerless Gait Analysis: Multi-Camera and Single-Camera Approaches in Comparison to Marker-Based Gait Analysis