Where Gait Recognition Breaks: A Four-Probe Diagnosis of Covariate Collapse

Gait recognition identifies people from how they walk, filmed at a distance and without their cooperation. On clean laboratory data it looks close to solved. It fails badly under covariates, meaning changes that have nothing to do with identity. A person wearing a heavy coat, carrying an umbrella, or seen from a steep overhead camera can be missed. On the Cross-Covariate Gait Recognition benchmark (CCGR), a strong recognizer scores 42.5% rank-1, and accuracy falls to about 2% from high-pitch surveillance-camera views (Cross-Covariate Gait Recognition: A Benchmark). The field answers each such drop with a new architecture: geometric equivariance, semantic body parsing, wavelet motion features, diffusion alignment. Nobody first asks where the identity signal was lost. It may be absent from the input pixels. It may be present in the input but destroyed by the network's features. Or it may survive in the features but sit in the wrong place, so that probe and gallery no longer line up.

Experimental physics has a standard answer to this kind of confusion. When physicists measure a sample's resistance inside a diamond anvil cell, the lead contacts change unpredictably under pressure. They therefore separate the path that drives current from the probes that read voltage (the four-probe method), so the sample's own resistance is not mixed with the contact artifacts. They also co-load a reference material with a known transition to calibrate the pressure scale (Superconductivity in HfTe5 across weak to strong topological insulator transition induced via pressures, Effects of pressure on electrical property of BaIrO3). The lutetium hydride episode shows what goes wrong without these controls: a sharp resistance drop that looked like room-temperature superconductivity turned out to be an ordinary metal-to-metal transition, and it reproduced in only about 30% of samples (Unveiling a Novel Metal-to-Metal Transition in LuH2). We propose to bring this measurement discipline to gait covariates. We borrow the procedure, not the physics.

Core Claim

Any accuracy drop under a covariate can be split, by independent probes, into four parts that add up exactly. The first is an input gap: identity information that is missing from the input itself. The second is a representation gap: information the frozen network's features discarded. The third is a readout gap: information that is present in the features but needs labeled data to extract. The fourth is an alignment gap: a shift of the feature distribution that a label-free transport map can undo.

Our central hypothesis is falsifiable. We expect a large share of the catastrophic collapses, including the overhead-camera 2%, to be alignment and readout gaps rather than input gaps. If so, a cheap, covariate-conditioned optimal-transport correction on frozen features will recover much of the loss with no new backbone. If the diagnosis instead shows mostly input gaps, that is equally useful: no architecture operating on silhouettes can fix the problem, and the field should change sensors or inputs instead.

Research Program

We fix a covariate condition $c$ (for example "overhead view, coat, umbrella") and a reference condition (normal walking, side view). We compute five rank-1 accuracies on identities disjoint from any data used to fit the probes.

$A_{\text{ref}}$ is accuracy in the reference condition. $A_0$ is the end-to-end accuracy of the unchanged recognizer under $c$; this is the "two-probe" number that benchmarks report today. $A_T$ is accuracy after we fit an optimal-transport map on unlabeled condition-$c$ features of the training identities and apply it to the probe features. An optimal-transport map is the lowest-cost way to move one set of feature points onto another. $A_P$ is accuracy after we train a small metric head on the frozen features using labeled condition-$c$ sequences from the training identities. $A_R$ is the best accuracy among several architectures retrained with condition-$c$ data included; it serves as our operational estimate of how much identity information the input contains. The total drop then splits as

$$ A_{\text{ref}} - A_0 = (A_{\text{ref}} - A_R) + (A_R - A_P) + (A_P - A_T) + (A_T - A_0), $$

giving, in order, the input, representation, readout and alignment gaps. The decomposition is only trustworthy if each probe is calibrated. We therefore add two known standards, which play the role of the co-loaded lead.

The positive standard is a synthetic condition that preserves identity: left-right mirrored and slightly rescaled reference sequences. Here the true input gap is zero by construction, so any input gap the protocol reports measures the probes' own bias. The negative standard randomly permutes identity labels within condition $c$. Here recoverable information is zero, so $A_T$, $A_P$ and $A_R$ must fall to chance. If they do not, the probes are leaking. We follow the lutetium hydride lesson and treat reproducibility as a quantity to measure. Every gap is reported across at least five seeds and several identity splits. A claimed recovery counts only if its confidence interval excludes zero.

The transport step takes its method from the covariate-shift optimal-transport literature. We use a covariate-aware formulation in which the transport cost is conditioned on the observed nuisance labels, following Tightening Causal Bounds via Covariate-Aware Optimal Transport. That paper conditions transport plans on covariates for causal bounds, not recognition. We also draw on the representation-level view of Optimal Representations for Covariate Shift. CCGR provides the nuisance labels we condition on: carrying type, clothing, terrain, speed and camera pitch. The conditional map lets a coat's shift be corrected differently from an overhead view's shift. It also lets compound conditions be tested for additivity: we check whether the map for "coat plus overhead" is close to the composition of the two single-covariate maps.

This program is mainly about measurement. Its contribution differs from the architectures it will diagnose. Learning Geometric Invariance for Gait Recognition hard-codes reflection, rotation and isotropic scaling. Cross-Covariate Gait Recognition: A Benchmark parses bodies from carried objects. Explicit Time-Frequency Dynamics for Skeleton-Based Gait Recognition adds wavelet features of joint velocity. DiffCrossGait aligns silhouettes and LiDAR through shared diffusion noise. Each of these papers reports a single end-to-end number, so none can say which gap it closed. The protocol can answer that question. For example, it can test whether RRS-Gait's gains come from shrinking the alignment gap (which a transport map would equally close) or the representation gap (which it would not). The disentangling models of Gait Recognition via Disentangled Representation Learning and Two-Point Gait try to separate motion from appearance inside the model. We instead separate the sources of failure from outside the model, which works with any backbone.

Baselines

The naive reference point is the unmodified end-to-end recognizer evaluated under each condition, exactly as CCGR reports it. For DeepGaitV2 on CCGR Hard, the reported value is 42.5% rank-1, and overhead camera views drop to about 2% (Cross-Covariate Gait Recognition: A Benchmark). On CASIA-B we use GaitSet's reported 95.0% normal-walking, 87.2% bag-carrying and 70.4% coat-wearing rank-1 (GaitSet). We rerun both in OpenGait (ShiqiYu/OpenGait) to confirm that we reproduce these numbers before measuring anything.

The strongest existing method on the headline benchmark is ParsingGait on a DeepGaitV2 backbone, reported at 52.7% rank-1 on CCGR Hard. It is the right competitor because it attacks the same confusion of body and carried object by changing the input. We rerun it with the same identity splits and the same number of training epochs. Our transport-corrected silhouette model beats it if it exceeds 52.7% on CCGR Hard, or matches it without requiring RGB parsing. Overhead views are the harder test. Recovering even 10 points above the roughly 2% floor would count as a substantive result, provided the negative standard stays at chance.

Moment-matching adaptation is the cheapest label-free rival to optimal transport. It re-estimates batch-normalization statistics per condition and matches feature means and covariances. It gets the same unlabeled target features and the same hyperparameter-search budget as the transport map. Full fine-tuning gets exactly the same labeled condition-$c$ sequences as the metric-head probe, which fixes the labeled-data budget.

The key ablation removes the covariate conditioning from the transport cost, giving an unconditional optimal-transport map. This tests whether knowing the nuisance labels is what makes the correction work.

Why not just…?

Why not just report per-condition accuracy, as CCGR already does? Per-condition accuracy tells you that the system failed under an overhead view, but not why. The 2% overhead number is consistent with three different stories: the pixels carry no identity, the features threw it away, or the features are fine but displaced. Each story implies a different remedy. The benchmark paper's own data cannot tell them apart, and our decomposition is designed to.

Why not just fine-tune on the target condition? This is a real threat, and we treat it as part of the protocol rather than an alternative to it: fine-tuning with labels is essentially how we measure $A_P$ and $A_R$. If full fine-tuning closes nearly everything, the conclusion stands and simply becomes practical: the failure was a readout or representation problem that labels fix. The open question is whether the label-free transport map closes a comparable share. If it does not, we will report that the cheap correction is insufficient rather than claim otherwise.

Why not just use larger models and heavier augmentation? Learning Geometric Invariance for Gait Recognition argues that data-driven models fail on unseen conditions, and DeepGaitV2's collapse on CCGR supports this for one large model. The claim is still untested for layer-wise large-vision-model features such as those in BiggerGait. We therefore include BiggerGait features as a backbone in the diagnosis. If its alignment gap is near zero, scale is the answer, and our transport step adds nothing for it.

Why not just use pose skeletons? Skeletons discard body-shape information, and pose estimators jitter under long coats, as noted by A Comprehensive Survey on Deep Gait Recognition and by the wavelet paper. We run the decomposition on SkeletonGait from OpenGait alongside silhouette models. This directly measures whether skeletons trade a smaller alignment gap for a larger input gap.

Proposed Experiments

The main study runs the decomposition on every CCGR single and compound condition, with overhead pitch analysed separately. Backbones are GaitSet, DeepGaitV2, SkeletonGait and BiggerGait, all in OpenGait. We add CASIA-B as a sanity check where published numbers anchor the reference. We use SUSTech1K to test whether the gaps change when LiDAR is the input; there, DiffCrossGait reports +9.3% for carrying and +8.5% for clothing over its baseline. Finally, Gait3D and GREW test whether the proportions of the four gaps hold outside the lab. No reported result was found on Lacuna for overhead-view accuracy on Gait3D or GREW, so those numbers will be new measurements.

Three checks decide the outcome. The first is calibration: the positive standard must show an input gap within seed noise of zero, and the negative standard must drive every probe to chance. If either fails, the decomposition is not reported. The second is the hypothesis test: under overhead views, the alignment and readout gaps together must account for more than half of the total drop. The third is additivity: on compound conditions, the composed single-covariate transport maps must predict the compound map's accuracy to within a few points. This tells us whether CCGR's "mixed covariates" are genuinely new failure modes or combinations of known ones.