Gait recognition identifies people by how they walk. Most systems first cut the person out of each video frame as a black-and-white shape, called a silhouette, and then feed the sequence of shapes to a neural network. Accuracy falls sharply when the camera angle changes, when lighting is poor, or when the subject carries something. The literature blames several culprits: bad foreground cut-outs, the limits of visible-light cameras, viewpoint changes, and the information discarded when a 3D body is flattened into a 2D image. These causes have never been separated, because each paper changes several of them at once.
We propose to separate them on one dataset whose design makes a controlled comparison possible. SUSTech1K records each walk with both an RGB camera and a LiDAR, a laser range sensor. Using the two together, we build a "ladder" of input representations. All rungs share the camera's image plane and the same network. Each rung changes exactly one thing: who did the segmentation (appearance in visible light, or 3D geometry from the LiDAR's own near-infrared laser), whether depth survives the flattening, whether absolute body size survives the usual resizing to a fixed box, and whether the model saw synthetic camera viewpoints during training. The accuracy difference between adjacent rungs measures what that factor costs. The final rung is also a practical method, a cheap 2D pipeline. If the ladder's diagnosis is right, it should approach the accuracy of point-cloud-native models.
Core Claim
We hypothesize that most of the accuracy gap between 2D silhouette pipelines and 3D point-cloud gait models comes from two removable choices rather than from projection itself. The first is appearance-based segmentation in the visible band. The second is the habit of normalizing every subject to a unit bounding box, which throws away height and breadth. A 2D encoder fed three things should therefore come close to a point-native 3D model at 2D cost:
- geometry-derived masks,
- per-pixel metric depth,
- a ground-anchored canvas at a fixed meters-per-pixel scale,
trained with virtual viewpoints rendered from the point cloud. If it does not, the remaining gap is the true price of projection. That would itself be a useful, previously unmeasured number.
The comparison targets come from the file. On SUSTech1K, LidarGait++ reports 92.7% rank-1 accuracy (the fraction of queries whose top match is the correct person) using raw points with size-aware encoding. LidarGait reports 86.77% with range-view depth maps against 64.70% with LiDAR silhouettes. The LidarGait++ paper describes its 5.9-point lead as over LidarGait, consistent with that 86.77% figure. The LidarGait source also quotes 91.80% "overall"; we will reconcile this against the released OpenGait configs before treating either number as the reference.
Prior Work and Its Role
Three groups of papers motivate the ladder.
- Gait models that each fix one cause, but confound several.
- LidarGait replaced camera silhouettes with LiDAR range-view depth maps. It also fused three fixed projections (MV-LidarGait). In doing so it changed sensor, segmentation, depth, and viewpoint together.
- LidarGait++ abandoned projection for raw points and reinjected absolute altitude. Its clothing-change accuracy rose from 58.7% to 79.7%. It does not say how much of that came from size awareness and how much from avoiding projection.
- LiCamGait fuses a range-image branch with a point branch through cross-attention and reaches 90.23% on SUSTech1K. It is a fusion system, not an attribution study.
- Methods that attack segmentation or viewpoint in 2D.
- SMPLGait uses fitted 3D body models to normalize silhouette features across views. It still depends on RGB silhouettes.
- DenoisingGait avoids binary masks by reading features out of a diffusion model at an intermediate noise level. It stays in 2D and costs a diffusion forward pass per frame.
Our ladder differs from both. It removes the segmentation dependence by changing the segmentation source, and it removes the viewpoint dependence by rendering from points rather than fitting a body model.
- Work that supplies the "sensing band" lens and the augmentation technique.
- Multi-spectral SIFT found near-infrared adds information beyond correlated visible channels.
- IDD-AW reports roughly 3% mIoU gains from adding NIR in adverse weather.
Both concern passive imaging. LiDAR return geometry is a different case, an active near-infrared measurement that ignores scene lighting and color camouflage. We treat it as the alternative band for segmentation. The augmentation technique, rotating point clouds and re-rendering depth from virtual cameras, comes from Multi-view Dynamic Images and SEIL. We reuse it rather than claim it.
- Constraining evidence.
- The umbrella failures in LidarGait and LiCamGait show that geometric segmentation has its own failure mode: objects get merged into the body.
- GaitSnippet and LidarGait's finding that set pooling (ignoring frame order) beats sequence models on LiDAR suggest temporal order is unreliable here.
We therefore fix order-invariant set pooling across all rungs rather than let temporal modeling become a confound.
Research Program
Every rung uses the same GaitBase encoder with set pooling and triplet-plus-cross-entropy loss from OpenGait, with identical training budget. The rungs are:
| Rung | Input | Factor changed from previous rung |
|---|---|---|
| R1 | Dataset's RGB silhouettes | (start: visible-band appearance segmentation) |
| R2 | LiDAR person cluster projected into the camera image, splatted into a soft mask | Segmentation source (projection held fixed) |
| R3 | R2 plus metric depth per pixel | Depth survives flattening |
| R4 | R3 rendered on a ground-anchored canvas at fixed meters-per-pixel instead of a cropped, resized box | Absolute size retained |
| R5 | R4 trained with depth renderings from randomly sampled virtual cameras (yaw over the full circle, pitch and height jittered); tested only on the real camera view | Synthetic viewpoints |
The difference R1 minus R2 isolates the segmentation source. R2 to R3 measures the depth discarded by flattening, R3 to R4 measures scale erasure, and R4 to R5 measures viewpoint overfitting.
For R2, the splatted soft mask has a few-pixel blur. It absorbs the small misregistration between sensors that the multispectral literature flags as a standing problem. R2 requires SUSTech1K's camera–LiDAR calibration. If the released calibration is unusable, we will estimate it by maximizing overlap between projected LiDAR clusters and RGB masks on well-lit sequences, and report the residual reprojection error. If the released point clouds keep per-point return intensity, R3 gains an intensity channel as a sub-ablation. That gives one more test of whether active-band reflectance carries identity beyond geometry.
We analyze every gap separately for each SUSTech1K condition: illumination, clothing, carrying, umbrella, and view. The diagnosis is predictive, not just descriptive:
- The R1-to-R2 gain should concentrate in poorly lit sequences.
- The R3-to-R4 gain should concentrate in clothing change, where shape cues blur but height does not.
- Umbrella sequences should hurt R2 through R5 through geometric merging. That would show the band switch trades one segmentation failure for another rather than solving segmentation.
Baselines
Naive reference. The do-nothing reference is R1 with standard 2D augmentation (crops, flips, small affine warps) and no LiDAR. This is the representation most deployed gait systems use. Every later rung is measured against it, so it is trained for the same steps and the same hyperparameter search as R5.
Strongest existing method: LidarGait++. LidarGait++ is the strongest competitor because it is exactly the "don't project at all" answer to our question, and it reports 92.7% rank-1 on SUSTech1K. We will rerun it from OpenGait on our machines with its released config. We will report both the reproduced and the published number, plus wall-clock training and inference cost. R5 counts as beating it if either condition holds:
- It matches it within the run-to-run spread of three seeds at clearly lower inference cost.
- It exceeds it on the clothing-change and cross-view subsets.
A gap of more than three points in LidarGait++'s favor will be read as the measured cost of projection, not as a failure to report.
Fixed multi-view projection. MV-LidarGait from LidarGait is the strongest projection-based competitor. It fuses three fixed views, so it tests whether continuous synthetic viewpoints add anything over a few hand-chosen ones. It gets the same encoder and budget as R5.
Fusion system. LiCamGait's HMRNet (90.23% on SUSTech1K) represents fusion of both modalities. We run it if its code is obtainable. Otherwise we cite its published number and mark it as not reproduced under our budget.
Mask-free RGB baseline. For the segmentation question we include DenoisingGait, run from OpenGait on SUSTech1K RGB. The file states it was evaluated there but gives no number, so: no reported result found on Lacuna for DenoisingGait on SUSTech1K. Its reported figure is 84.0% rank-1 in CCPG's cloth-changing setting, which is not comparable here.
Ablation of the key ingredient. The key ingredient is the ground-anchored metric canvas. We run R4 and R5 with ordinary box normalization but with body height appended as a single scalar to the final embedding. This checks whether scale needs to live in the image or just in the descriptor.
Why not just…?
Why not just use LidarGait++ and stop? If the only goal were SUSTech1K accuracy, that would be the right answer, and our experiments may show it. The reason to go further is that LidarGait++ cannot say why it wins, and it needs neighborhood queries over points that cost more than 2D convolutions. The ladder measures whether its advantage lies in avoiding projection or in keeping size. If R5 lands within seed noise of 92.7%, the field gets a cheaper pipeline. If R5 does not, the field gets the first direct measurement of what projection costs.
Why not just append body height to a standard silhouette model? This is a real threat to the canvas idea, which is why the scalar-height ablation exists. If the scalar recovers most of the R3-to-R4 gain, we will say so. The conclusion would then be that scale erasure is cheaply fixable at the descriptor level, and the canvas becomes unnecessary.
Why not just get better RGB masks, or skip masks as DenoisingGait does? Better appearance-based segmentation still lives in the visible band, which fails under darkness and color camouflage. DenoisingGait removes the binary threshold but not the dependence on visible-light appearance, and it adds diffusion cost. The R1-versus-R2-versus-DenoisingGait comparison on the illumination subset settles this. If DenoisingGait matches R2 in low light, the band switch adds nothing beyond removing hard thresholds.
Why not just add rotation augmentation to the point-cloud model? Rotating raw points is the natural 3D analogue of R5 and could erase R5's advantage. We will train LidarGait++ with the same virtual-viewpoint sampling. If it gains as much as R5 does over R4, viewpoint augmentation is model-agnostic. The comparison then reduces to the projection question alone.
Proposed Experiments
Main experiment. We train all five rungs plus the baselines on the SUSTech1K training split with three seeds each. We report rank-1 overall and per condition, and inference time per sequence. The headline figure is a stacked decomposition of the R1-to-R5 gain into segmentation, depth, scale, and viewpoint components, with confidence intervals across seeds.
Out-of-distribution transfer. We train on SUSTech1K and test on the LiCamGait benchmark without fine-tuning. Published in-domain references there are HMRNet at 80.76% and LidarGait at 74.15%, with silhouette methods at 30.31% versus HMRNet's 68.47% in cross-view. FreeGait is a further target if accessible, where no reported result was found on Lacuna.
Decisive checks. The diagnosis would be falsified by any of the following:
- If R1-to-R2 is flat on low-light sequences, the sensing-band account of silhouette fragility is wrong for this data.
- If R5 trails LidarGait++ by more than three points even after both receive the same viewpoint augmentation, projection itself is the dominant loss, and the 2D route should be abandoned.
- If R5 transfers to LiCamGait no better than R1, the gains are dataset-specific.
Scope. We do not test passive thermal or shortwave infrared, because no gait benchmark with those bands appears in the file. Extending the ladder to a sensor rig with thermal cameras is the natural next step once the method is validated.
References
- LidarGait++: Learning Local Features and Size Awareness from LiDAR Point Clouds for 3D Gait Recognition
- LIDAR GAIT: Benchmarking 3D Gait Recognition with Point Clouds
- Gait Recognition in Large-scale Free Environment via Single LiDAR
- Gait Recognition in the Wild with Dense 3D Representations and A Benchmark
- On Denoising Walking Videos for Gait Recognition
- Multi-spectral SIFT for scene category recognition
- IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather
- Action recognition for depth video using multi-view dynamic images
- SEIL: Simulation-augmented Equivariant Imitation Learning
- GaitSnippet: Gait Recognition Beyond Unordered Sets and Ordered Sequences