3D Local Convolutional Neural Networks for Gait Recognition

Zhen Huang, Dixiu Xue, Xu Shen, Xinmei Tian, Houqiang Li, Jianqiang Huang, Xian-Sheng Hua

Introduction

Gait recognition—identifying individuals by their unique walking patterns—is a powerful biometric tool that operates at a distance without requiring subject cooperation. Despite its potential, it remains a challenging computer vision problem due to variations in clothing, carrying conditions, and camera angles, all of which can drastically alter a person's silhouette.

Researchers from the University of Science and Technology of China and Alibaba Group have proposed a novel architecture: 3D Local Convolutional Neural Networks. This approach shifts from rigid, global feature extraction to an adaptive, part-based method that "zooms in" on specific body parts across both space and time.

The Limitations of Fixed Partitioning

Traditional gait recognition models typically fall into two categories: global methods, which analyze the entire silhouette, and part-based methods, which divide the body into fixed horizontal stripes. The authors identify three fundamental flaws in fixed-stripe partitioning:

  1. Spatial Overlap: Moving parts, such as an arm swinging in front of the torso, often occupy the same horizontal stripe.
  2. Temporal Displacement: A body part may move between different stripes over the course of a gait cycle.
  3. Scale and Frequency: Different body parts move at varying scales and frequencies; for instance, the head remains relatively stable while legs move rapidly over larger distances.

3D Local CNN Architecture

To address these issues, the proposed model utilizes a 3D Local Operation block designed to track and analyze specific body parts dynamically.

Comparison of convolution types: (a) 2D convolution, (b) 3D convolution, (c) Non-local block, and (d) the proposed 3D local CNN which adaptively samples spatio-temporal volumes.

The architecture consists of four primary modules:

  • Localization Module: A sub-network that predicts seven parameters (center, scale, and temporal length) for each of the six defined body parts (head, torso, and four limbs).
  • Sampling Module: Extracts 3D "patches" from the feature map using a differentiable attention mechanism, allowing for end-to-end training via backpropagation.
  • Feature Extraction (FE): Dedicated 3D convolution layers process the sampled local volumes to capture part-specific movement patterns.
  • Fusion Module: Integrates local part features with global features using a $1 \times 1 \times 1$ convolution to produce a final refined feature vector.

Experimental Evaluation

The model was evaluated on two major benchmarks:

  • CASIA-B: Contains 124 subjects under normal walking (NM), carrying a bag (BG), and changing clothes (CL) conditions.
  • OU-MVLP: The largest public gait dataset, featuring over 10,000 subjects across 14 view angles.

Key Results

The 3D Local CNN achieved state-of-the-art (SOTA) performance:

  • CASIA-B (Clothing): On the challenging "CL" subset, the model reached 83.7% accuracy, significantly outperforming previous SOTA models like GaitPart (78.7%) and GLN (77.5%).
  • OU-MVLP: Achieved a mean rank-1 accuracy of 90.9%, surpassing the previous best of 89.2%.

Discussion and Insights

A key finding is that the localization module learns to track body parts in an unsupervised manner; the model was not provided with ground-truth bounding boxes for limbs but learned to locate them to minimize identification loss. Furthermore, the model's robustness to clothing changes stems from its ability to focus on the underlying movement of limbs even when the global body shape is obscured.

Ablation studies confirmed that performance improves as more local parts are added. The researchers also noted that inserting these local blocks into deeper CNN layers is most effective, as features at these levels are more semantically meaningful.

Future Work

While the 3D Local CNN represents a significant advancement, several areas for future research remain. The current model relies on a predefined number of parts and initial positions. Future iterations could explore systems that automatically discover optimal "interest zones." Additionally, optimizing the computational complexity for real-time edge device deployment and exploring more sophisticated feature extraction modules for local paths are promising directions.

Summary updated

Abstract

The goal of gait recognition is to learn the unique spatiotemporal pattern about the human body shape from its temporal changing characteristics. As different body parts behave differently during walking, it is intuitive to model the spatio-temporal patterns of each part separately. However, existing part-based methods equally divide the feature maps of each frame into fixed horizontal stripes to get local parts. It is obvious that these stripe partition-based methods cannot accurately locate the body parts. First, different body parts can appear at the same stripe (e.g., arms and the torso), and one part can appear at different stripes in different frames (e.g., hands). Second, different body parts possess different scales, and even the same part in different frames can appear at different locations and scales. Third, different parts also exhibit distinct movement patterns (e.g., at which frame the movement starts, the position change frequency, how long it lasts). To overcome these issues, we propose novel 3D local operations as a generic family of building blocks for 3D gait recognition backbones. The proposed 3D local operations support the extraction of local 3D volumes of body parts in a sequence with adaptive spatial and temporal scales, locations and lengths. In this way, the spatio-temporal patterns of the body parts are well learned from the 3D local neighborhood in partspecific scales, locations, frequencies and lengths. Experiments demonstrate that our 3D local convolutional neural networks achieve state-of-the-art performance on popular gait datasets. Code is available at: https://github. com/yellowtownhz/3DLocalCNN.
BibTeX
@inproceedings{huang2021d,
  author = {Zhen Huang and Dixiu Xue and Xu Shen and Xinmei Tian and Houqiang Li and Jianqiang Huang and Xian-Sheng Hua},
  title = {{3D Local Convolutional Neural Networks for Gait Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01465},
}
Figure