GaitSet: Cross-View Gait Recognition Through Utilizing Gait As a Deep Set

Hanqing Chao, Kun Wang, Yiwei He, Junping Zhang, Jianfeng FengIEEE TPAMI 2021 · OpenReview

Abstract

Gait is a unique biometric feature that can be recognized at a distance, offering broad applications in crime prevention and forensic identification. Traditional gait recognition methods rely on either templates, which struggle to preserve temporal information, or sequences, which impose rigid constraints that reduce flexibility. This paper introduces GaitSet, a novel perspective that treats gait as a "deep set." By integrating a set of gait frames through a global-local fused deep network, the model becomes immune to frame permutations and can naturally integrate frames from diverse scenarios. Experiments demonstrate state-of-the-art performance, achieving 96.1% rank-1 accuracy on CASIA-B and 87.9% on OU-MVLP. Notably, GaitSet remains highly robust under complex conditions like coat-wearing and can identify individuals with as few as seven random frames.

Introduction

Gait recognition—identifying individuals by their walking style—is a powerful biometric because it works at a distance without requiring the subject's cooperation. However, traditional methods often struggle with changes in viewpoint, clothing, or carrying conditions.

Existing approaches generally fall into two categories: gait templates and gait sequences. Gait templates, like Gait Energy Images (GEI), compress a walking cycle into a single image. While computationally efficient, they lose vital temporal and fine-grained spatial information. Conversely, gait sequences treat gait as a video, using 3D convolutions or Recurrent Neural Networks (RNNs). While these preserve temporal data, they are often "brittle," requiring specific frame rates and continuous sequences.

Gait silhouettes representing the input to the GaitSet model.

The authors of GaitSet propose a paradigm shift: stop treating gait as a rigid video sequence and start treating it as an unordered set of independent silhouettes. The key insight is that while walking is periodic, the unique set of silhouettes a person produces is more discriminative than the specific order of their steps.

Methods: The GaitSet Architecture

The GaitSet architecture follows a three-step pipeline designed to extract and aggregate features from a "bag of silhouettes."

The GaitSet architecture, featuring Frame-level Feature Extraction, Set Pooling (SP), and Horizontal Pyramid Mapping (HPM).

1. Frame-level Feature Extraction

A standard 2D Convolutional Neural Network (CNN) processes each silhouette in the input set independently. This stage extracts "local" spatial features from every frame before any temporal or set-level aggregation occurs.

2. Set Pooling (SP)

This is the core innovation of the model. A Set Pooling operation (such as max, mean, or a learned attention mechanism) aggregates all frame-level features into a single "set-level" feature map. Because these operations are commutative, the model is permutation invariant—scrambling the input frames does not change the output. This allows the model to handle non-consecutive frames and varying frame rates seamlessly.

3. Horizontal Pyramid Mapping (HPM)

To capture both global body shape and local limb movements, the set-level feature map is split into horizontal strips at multiple scales. Each strip is processed by an independent fully connected layer to create a final, highly discriminative feature vector. This ensures that the model pays attention to specific body parts (like the swing of an arm or the stride of a leg) while maintaining a global view of the subject.

Additionally, a Multilayer Global Pipeline (MGP) pulls features from different depths of the CNN, ensuring the model captures both fine-grained details from shallow layers and abstract patterns from deeper layers.

Experimental Results

The model was evaluated on the two most prominent gait recognition benchmarks: CASIA-B (124 subjects, 11 angles, 3 conditions) and OU-MVLP (over 10,000 subjects).

  • State-of-the-Art Accuracy: On CASIA-B, GaitSet reached 96.1% rank-1 accuracy under normal conditions. In the challenging "Coat" condition, where body shape is obscured, it achieved 70.3%, outperforming previous best methods by over 15%.
  • Large-Scale Performance: On the massive OU-MVLP dataset, it achieved an average accuracy of 87.9%.
  • Efficiency with Sparse Data: The model is remarkably robust to low frame counts. It maintains 85.0% accuracy on CASIA-B using only 7 frames per sequence, whereas traditional template-based methods often fail with such limited data.

Conclusion

GaitSet demonstrates that order invariance is a feature, not a bug, in gait recognition. By treating frames as a set, the model gains immense flexibility and robustness against missing data or environmental changes. While clothing remains a significant challenge for the field, the "Deep Set" perspective offers a promising path forward for video-based biometric tasks where temporal order may be less critical than the cumulative information within the set.

Summary updated

Abstract

Gait is a unique biometric feature that can be recognized at a distance; thus, it has broad applications in crime prevention, forensic identification, and social security. To portray a gait, existing gait recognition methods utilize either a gait template which makes it difficult to preserve temporal information, or a gait sequence that maintains unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper, we present a novel perspective that utilizes gait as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deep set</i> , which means that a set of gait frames are integrated by a global-local fused deep network inspired by the way our left- and right-hemisphere processes information to learn information that can be used in identification. Based on this <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deep set</i> perspective, our method is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">immune to frame permutations</i> , and can naturally <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">integrate frames from different videos</i> that have been acquired under different scenarios, such as diverse viewing angles, different clothes, or different item-carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 96.1 percent on the CASIA-B gait dataset and an accuracy of 87.9 percent on the OU-MVLP gait dataset. Under various complex scenarios, our model also exhibits a high level of robustness. It achieves accuracies of 90.8 and 70.3 percent on CASIA-B under bag-carrying and coat-wearing walking conditions respectively, significantly outperforming the best existing methods. Moreover, the proposed method maintains a satisfactory accuracy even when only small numbers of frames are available in the test samples; for example, it achieves 85.0 percent on CASIA-B even when using only 7 frames. The source code has been released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/AbnerHqC/GaitSet</uri> .
BibTeX
@article{chao2021gaitset,
  author = {Hanqing Chao and Kun Wang and Yiwei He and Junping Zhang and Jianfeng Feng},
  title = {{GaitSet: Cross-View Gait Recognition Through Utilizing Gait As a Deep Set}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=QFteIYw1Bo},
}
Figure