Video-based gait assessment now has two main designs, and each discards something. The first reduces every recording to a bare skeleton or body mesh, which survives a change of camera but throws away everything visible on the body. The second feeds whole frames to a video classifier, which keeps the pixels but learns the camera, the room, and the viewpoint along with the patient. This proposal tests a middle route borrowed from cell biology. A pose estimator, a network that finds joints in each frame, marks where each limb segment is. We pool appearance features only inside those limb regions, much as cell-profiling software measures texture only inside segmented nuclei and cytoplasm. We then remove the camera's signature by comparing each new site's recordings of unaffected people against a reference, after matching frames by body pose. Paired captures of the same people on several cameras are used only to check the correction, never to fit it.
The question is concrete. Does pixel information inside body regions add diagnostic signal beyond the skeleton once camera effects are removed? And can the camera effect be estimated without paired recordings or diagnostic labels at the new site? If the answer to the first question is no, that result is also worth having: it would show that skeleton-only representations lose nothing on this task.
Core Claim
Skeletal geometry is a camera-robust anchor. Appearance features pooled inside pose-defined limb masks depend jointly on three things: the camera, the person's pose at that instant, and the person's health. Recordings of healthy people are a site-level negative control, playing the same role as untreated control wells on a microscopy plate. If we estimate a site's appearance shift from those control frames conditioned on pose, we can remove the camera signature while keeping disease-related appearance and kinematic differences.
The claim has two parts, and each can be falsified on its own. The first part is that pose-masked pooling isolates body appearance from room and background. The second is that pose-conditioned, control-anchored normalization beats both unconditional normalization and adversarial feature removal at keeping diagnostic signal under camera change.
Why this gap exists
The motivating evidence comes from both sides of the trade-off. CARE-PD pools gait data from eight clinical centres and handles site shift by converting everything to textureless SMPL body meshes (a standard parametric human body model). Its cross-site protocols make generalization measurable, but no appearance cue can survive that conversion. GAVD trains whole-video action recognizers on heterogeneous clinical and internet footage. These reach 94% test accuracy and 87% on the laboratory CASD set, but fall to 56–66% on side views against 91–97% on front and back views. That gap shows how much whole-frame models depend on recording conditions. Validation studies such as Park et al. and VideoRun2D establish that markerless joint angles can approach the roughly 5-degree clinical tolerance. That tells us the skeleton itself is a usable anchor.
The method comes from three other literatures. Mask-weighted pooling, which averages deep features only over a segmented region, improved CT triage specificity by 2.0 points over plain global pooling in Zhang et al.. In Lesion-Centric Latent Phenotypes it carried breast-ultrasound features across scanners to an external AUC of 0.982. That work is the closest prior art. It uses static images, masks from a segmentation network, and calibration by fine-tuning bottleneck layers on labelled target data. We instead obtain masks from pose in video and calibrate without target labels or retraining.
The normalization logic comes from morphological profiling. Cell Painting pipelines (Caicedo et al.) compute features per cellular compartment, and normalizing against negative controls is standard practice. Set-DINO (Set-DINO) reaches 46.9% replicate recall on CORUM against 35.6% for engineered features. It does so partly through control-based z-scoring and cross-batch sampling, and we borrow its replicate-retrieval evaluation directly.
A constraining result comes from Degel et al.. There, adversarial vendor-blinding plus a shape prior cut cross-vendor surface error from 3.56 mm to 2.01 mm. This shows adversarial removal works when geometry is the target. But profiling's readouts are themselves texture, so adversarial stripping is exactly the tool most likely to erase them. Rastogi et al. add a second constraint. Their cross-camera fundus translation reached R² of 78.5%, but it required paired captures, and fine geometric biomarkers such as tortuosity transferred poorly. We take this as a warning that global distribution matching loses fine-scale structure.
Research Program
We will build a two-stream representation per video.
The skeletal stream uses camera-frame 3D joint angles, following Natraj et al. in avoiding world-coordinate alignment. A motion encoder pretrained on CARE-PD sequences embeds these angles.
The appearance stream starts from 2D keypoints and draws soft, dilated Gaussian masks around each limb segment: shank, thigh, foot, trunk, and forearm. The softness and dilation are deliberate. They tolerate keypoint jitter, and they keep a margin of surrounding context, which addresses the known failure of hard ROI masks that discard peri-structure cues. A frozen image encoder's feature maps are mask-weighted pooled per limb. This gives one appearance profile per body part per frame, structured like a compartment-wise cell profile.
The camera correction is fit at each site $s$ from a small set of recordings of unaffected walkers. Clinics can produce these cheaply, for example from staff or companions. For limb $b$, let $x$ be a frame's pooled appearance vector and $q$ its joint-angle vector. The corrected feature is
$$\tilde{x} = \frac{x - \mu_{s,b}(q)}{\sigma_{s,b}(q)},$$
where $\mu_{s,b}(q)$ and $\sigma_{s,b}(q)$ are the mean and spread of control-frame features at site $s$, limb $b$. Each control frame is weighted by a kernel on its pose distance to $q$. This is control-well z-scoring with pose as the matching covariate. It needs no paired data, no target diagnoses, and no gradient updates. A diagnostic head is then trained on the concatenated skeletal and corrected appearance embeddings at source sites.
The analysis has three layers.
The first is a variance decomposition of masked appearance features into camera, pose, subject, and diagnosis components. It determines whether pose matching is even needed. If pose explains little appearance variance, unconditional control normalization suffices.
The second is cross-camera generalization on clinical benchmarks.
The third is a check of what is removed. The correction should leave subject identity and diagnosis recoverable while making the camera unpredictable.
Baselines
The naive reference is a whole-frame video classifier, SlowFast as benchmarked on GAVD. It will be trained on identical splits with the same epoch budget. Its reported figures (94% GAVD test accuracy, 87% on CASD, 56–66% on side views) serve as sanity anchors. We will re-run it under our leave-source-out splits rather than quote it across protocols.
The strongest skeleton-only competitor is the motion encoder used in CARE-PD on pose alone. On reconstruction, CARE-PD reports 7.5 mm MPJPE (mean per-joint position error) for fine-tuned MotionAGFormer against 60.7 mm for the healthy-data pretrained model. For appearance-augmented cross-camera gait classification, no reported result was found on Lacuna. Beating the skeleton stream means the two-stream model improves held-out-site accuracy over skeleton-only, with confidence intervals across held-out sources that exclude zero.
The strongest appearance-side competitor is the lesion-centric recipe (Lesion-Centric Latent Phenotypes): mask-weighted pooling with target bottleneck fine-tuning. We give it the same small target budget, but as labelled diagnostic examples, since that is what its calibration needs.
The domain-adaptation baselines are three. The first is an adversarial camera discriminator in the style of Degel et al.. The second is ComBat and Harmony applied to pooled features, which CellPainTR beat on JUMP with a score of 0.60 against 0.54 and 0.56. The third is heavy colour, noise, and blur augmentation.
The key ablations remove one ingredient at a time. One drops the masks, pooling over the whole frame. One replaces pose-conditioned with unconditional control normalization. One replaces control normalization with unconditional Wasserstein marginal alignment in the style of Rastogi. All variants share the frozen encoder, the hyperparameter search budget, and the control-recording budget.
Why not just…?
Why not just keep the skeleton, as CARE-PD does? This is the most serious alternative, and it is partly the hypothesis under test. Skeletons are robust by construction. If the corrected appearance stream adds nothing at held-out sites, we will report that skeleton-only representations are sufficient for these tasks. The experiment settles it directly, and we treat that outcome as a real result rather than a failure.
Why not just z-score against healthy controls without pose matching? This is a real threat too, because it is exactly what profiling pipelines do with control wells. Pose matching matters only if limb appearance varies strongly with limb orientation, through foreshortening and clothing folds, and if controls and patients walk differently. The variance decomposition and the conditional-versus-unconditional ablation decide this. If pose explains little variance, we drop the matching and the contribution reduces to pose-masked profiling with control normalization.
Why not just train an adversarial camera discriminator? Degel et al. show adversarial removal works for segmentation geometry. Our targets, however, include texture, which is precisely what such a discriminator penalizes. We will measure diagnostic accuracy and subject retrieval after each correction. Adversarial removal should drive camera predictability to chance at a greater cost to diagnosis than control normalization does. If it does not, it wins.
Why not just augment or fine-tune on the new site? Rastogi et al. report that even a large foundation encoder kept device drift that needed explicit translation, which suggests augmentation alone falls short. Fine-tuning requires labelled target cases, which is the resource clinics lack. Both are run as baselines under matched budgets.
Proposed Experiments
The benchmark experiments use GAVD with videos grouped by source as a proxy for capture hardware, under leave-source-out evaluation, and CASD as an external laboratory domain. All results are stratified by viewpoint so that viewpoint shift is not credited to camera correction. CARE-PD supplies pretraining and cross-site reference for the skeletal stream only, since its meshes contain no pixels.
Because neither dataset isolates hardware, we will also record a small paired set. A modest cohort of healthy volunteers and volunteers with simulated gait abnormalities will walk while filmed at the same time by a phone, a webcam, and a laboratory camera from fixed views. These pairs are used only for evaluation. They let us measure, per limb feature, how much camera-attributable variance each correction removes. They also support subject replicate retrieval, measured as mean average precision for retrieving the same walk across cameras, and abnormality retrieval.
The decisive checks are three. After correction, a linear probe should fail to predict the camera on the paired set. Cross-camera retrieval of the same subject should rise relative to the uncorrected and unconditional variants. The two-stream model should beat skeleton-only at held-out GAVD sources and on CASD.
Two findings would count against the proposal. If pose-conditioned correction does no better than plain control z-scoring, the pose-matching part is unnecessary. If no appearance variant beats the skeleton, the appearance stream adds nothing to gait diagnosis. Either way we would report that directly.
References
- CARE-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson's Disease Gait Assessment
- Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset
- Verification of Markerless Gait Analysis: Multi-Camera and Single-Camera Approaches in Comparison to Marker-Based Gait Analysis
- VideoRun2D Demo: Markerless Body Tracking for Biomechanical Analysis of Running
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Data-analysis strategies for image-based cell profiling
- Weakly Supervised Set-Consistency Learning Improves Morphological Profiling of Single-Cell Images
- Domain and Geometry Agnostic CNNs for Left Atrium Segmentation in 3D Ultrasound
- Feature Space Translation Framework for Cross-Device Alignment and Derivation of Clinically Relevant Retinal Biomarkers from Foundation Model Embeddings
- 3D pose estimation for scalable remote gait kinematics assessment
- CellPainTR: Generalizable Representation Learning for Cross-Dataset Cell Painting Analysis