Hierarchical Spatio-Temporal Representation Learning for Gait Recognition

Lei Wang, Bo Liu

Abstract

Gait recognition is a biometric technique that identifies individuals by their unique walking styles, suitable for unconstrained environments and has a wide range of applications. While current methods focus on exploiting body part-based representations, they often neglect the hierarchical dependencies between local motion patterns. In this paper, we propose a hierarchical spatio-temporal representation learning (HSTL) framework for extracting gait features from coarse to fine. Our framework starts with a hierarchical clustering analysis to recover multi-level body structures from the whole body to local details. Next, an adaptive region-based motion extractor (ARME) is designed to learn region-independent motion features. The proposed HSTL then stacks multiple ARMEs in a top-down manner, with each ARME corresponding to a specific partition level of the hierarchy. An adaptive spatio-temporal pooling (ASTP) module is used to capture gait features at different levels of detail to perform hierarchical feature mapping. Finally, a frame-level temporal aggregation (FTA) module is employed to reduce redundant information in gait sequences through multi-scale temporal downsampling. Extensive experiments on CASIA-B, OUMVLP, GREW, and Gait3D datasets demonstrate that our method outperforms the state-of-the-art while maintaining a reasonable balance between model accuracy and complexity.

Introduction

Gait recognition—identifying people by how they walk—is a powerful biometric because it works at a distance without requiring the subject's cooperation. However, it faces two major hurdles. First, most current "part-based" models simply slice a person's silhouette into horizontal strips, ignoring the natural hierarchy of the body (e.g., feet are part of legs, which are part of the lower body) and the fact that different regions have vastly different motion dynamics. Second, gait videos often contain many nearly identical frames, creating temporal redundancy that standard models struggle to compress effectively without losing subtle motion cues.

The HSTL Framework

The proposed Hierarchical Spatio-Temporal Representation Learning (HSTL) framework processes gait sequences from "coarse to fine" through several novel modules designed to capture the complex dynamics of human movement.

The HSTL pipeline featuring hierarchical clustering, ARME, ASTP, and FTA modules.

Hierarchical Clustering

Instead of arbitrary slicing, HSTL uses hierarchical clustering as a pre-processing step to identify semantic body regions based on motion patterns. This recovers a multi-level structure: Level 1 represents the whole body, Level 2 divides it into upper and lower sections, and Level 4 reaches specific parts like the head, torso, legs, and feet. This motion-driven partitioning treats functional units (like thighs and calves) together before breaking them down further.

Adaptive Motion Extraction and Pooling

The Adaptive Region-based Motion Extractor (ARME) uses non-shared 3D convolutions for different body regions. Because the physics of a swinging arm differs significantly from a planted foot, the model learns unique filters for each region. The Adaptive Spatio-Temporal Pooling (ASTP) module then maps features from different levels of the hierarchy into a unified embedding, ensuring that both global "whole-body" features and fine-grained "part-specific" features contribute to the final identity signature.

Frame-level Temporal Aggregation (FTA)

To handle temporal redundancy, the FTA module uses multi-scale temporal pooling (with 3x1x1 and 5x1x1 kernels) to downsample the sequence. An attention-like weighting mechanism decides which frames are most informative for recognition, effectively compressing the sequence while preserving critical motion cues.

Experimental Evaluation

The model was evaluated on four major gait benchmarks: CASIA-B (controlled), OUMVLP (large-scale), GREW (in-the-wild), and Gait3D (unconstrained indoor).

HSTL consistently outperformed previous state-of-the-art (SOTA) methods across these datasets. On CASIA-B, it achieved 98.1% accuracy in normal walking and 88.9% in the challenging "Clothing" (CL) condition, demonstrating robustness to variations in appearance. On the GREW "in the wild" benchmark, HSTL reached 62.72% Rank-1 accuracy, a significant jump from the previous SOTA of 55.32%. Additionally, the model showed improved performance on children's gait, addressing a common bias toward adult walking patterns in previous models.

Accuracy vs. FLOPs comparison showing HSTL's superior balance between performance and complexity.

As shown in the Accuracy vs. FLOPs comparison, HSTL occupies a "sweet spot" in the design space, providing higher accuracy than competitors like GaitGL and MetaGait while maintaining lower computational complexity.

Conclusion

The HSTL framework demonstrates that modeling the human body as a hierarchy of moving parts significantly improves gait recognition performance. By combining adaptive motion extraction with multi-scale temporal aggregation, it achieves a superior balance between accuracy and efficiency. While challenges remain in unconstrained "in-the-wild" environments due to occlusion and low resolution, the hierarchical approach provides a robust foundation for future biometric systems.

Summary updated

Abstract

Gait recognition is a biometric technique that identifies individuals by their unique walking styles, which is suitable for unconstrained environments and has a wide range of applications. While current methods focus on exploiting body part-based representations, they often neglect the hierarchical dependencies between local motion patterns. In this paper, we propose a hierarchical spatio-temporal representation learning (HSTL) framework for extracting gait features from coarse to fine. Our framework starts with a hierarchical clustering analysis to recover multi-level body structures from the whole body to local details. Next, an adaptive region-based motion extractor (ARME) is designed to learn region-independent motion features. The proposed HSTL then stacks multiple ARMEs in a top-down manner, with each ARME corresponding to a specific partition level of the hierarchy. An adaptive spatio-temporal pooling (ASTP) module is used to capture gait features at different levels of detail to perform hierarchical feature mapping. Finally, a frame-level temporal aggregation (FTA) module is employed to reduce redundant information in gait sequences through multi-scale temporal downsampling. Extensive experiments on CASIA-B, OUMVLP, GREW, and Gait3D datasets demonstrate that our method outperforms the state-of-the-art while maintaining a reasonable balance between model accuracy and complexity.
BibTeX
@inproceedings{wang2023hierarchical,
  author = {Lei Wang and Bo Liu},
  title = {{Hierarchical Spatio-Temporal Representation Learning for Gait Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=SNsB74SmfP},
}
Figure