CTrGAN: Cycle Transformers GAN for Gait Transfer

Shahar Mahpod, Noam Gaash, Hay Hoffman, Gil Ben-ArtziWACV 2023

TL;DR

This paper introduces CTrGAN, a novel Transformer-based GAN for gait transfer from unconstrained videos. It addresses the challenge of transferring a target's characteristic gait onto a source's motion without requiring paired data or retraining for new sources, achieving significantly more realistic results than existing methods.

What Problem Does This Paper Address?

Gait transfer aims to replace a person in a video with another person, making the new person perform the original actions but with their own characteristic gait. Existing methods often struggle with unconstrained "in-the-wild" videos, require paired data (which is hard to obtain), or are limited to imitating the source's exact movements rather than transferring the target's style of movement. This means that even if a person's appearance is changed, their gait might still be recognizable as the original source's, failing to achieve true gait transfer. The paper tackles the problem of achieving realistic gait transfer from unconstrained videos, ensuring the transferred gait is identifiable as the target's, and developing a method to evaluate this transfer effectively.

Methods

The core of the proposed method is CTrGAN (Cycle Transformers GAN). It leverages the power of Transformers, which have shown great success in sequence-to-sequence tasks like natural language processing. Unlike previous approaches that focused on spatial attention between image patches, CTrGAN employs temporal attention between consecutive frames. This allows it to capture the dynamic nature of walking and transfer motion patterns effectively.

Key components of CTrGAN include:

  • Cycle-based Architecture: Similar to CycleGAN, it uses a cyclic process to ensure that translating from source to target and back to source results in the original source. This helps in learning robust mappings without paired data.
  • Transformers for Pose Translation: CTrGAN uses two Transformer-based generators: one to translate source poses to target poses ($G_{s \to t}$) and another to do the reverse ($G_{t \to s}$). These Transformers process sequences of poses, attending to temporal relationships.
  • Pose-to-Appearance Module: After generating the target's pose, a separate module renders the appearance of the target onto this pose. The paper mentions using an existing state-of-the-art approach for this.
  • Novel Key Selection: The method uses a K-means clustering approach on VGG features to select representative "keys" from the target's gait sequences, which are then used by the Transformer to guide the pose translation.
  • Temporal Attention: The attention mechanism operates in the temporal domain, focusing on how poses evolve over time, which is crucial for capturing gait dynamics.

Datasets

The primary dataset used for training and evaluation is CASIA-A. This dataset is a widely-used gait recognition benchmark containing video sequences of twenty subjects captured from multiple viewpoints. The authors also mention using a pre-trained VGG model for feature extraction, which was likely trained on ImageNet.

Key Insights and Ideas

  • Temporal Attention is Key for Gait: The paper strongly suggests that focusing on temporal relationships between frames, rather than just spatial relationships within frames, is critical for realistic gait transfer. This allows the model to learn and replicate the dynamic patterns of walking.
  • Transformers Excel at Sequence-to-Sequence Gait: Adapting Transformer architectures, known for their success in NLP, to handle sequences of poses for gait transfer proves highly effective.
  • Gait Recognition as an Evaluation Metric: Using established gait recognition models (GaitSet, GaitGL, GaitPart) to evaluate the quality of the transferred gait is a novel and insightful approach. It directly measures whether the generated gait is identifiable as the target's.
  • Cycle Consistency for Unpaired Data: The cyclic nature of the GAN architecture is essential for learning meaningful gait transfer without requiring perfectly aligned source-target video pairs.

Key Results

  • Significant Improvement in Target-Accuracy: CTrGAN achieves a substantial improvement in target-accuracy, meaning the generated gait is much more likely to be recognized as the target's. For example, when combined with V2V for pose-to-appearance, the target-accuracy is improved by approximately 21 times compared to existing methods.
  • Realistic Gait Generation: The generated gaits are described as an order of magnitude more realistic than those produced by previous methods.
  • Generalization to Unseen Sources: The model can transfer the target's gait from unseen sources, demonstrating its ability to generalize beyond the training data.
  • Effective Appearance Quality: While the primary focus is gait, the appearance quality metrics (SSIM, LPIPS, FID, IS) show comparable or slightly improved results when CTrGAN is used to generate poses before rendering.
  • High Accuracy Detector: The paper also proposes an appearance-based detector for generated videos, achieving an impressive 96.2% average detection accuracy.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while appearance quality is comparable, the main contribution of CTrGAN is in pose generation. Further improvements in the rendering mechanism itself are not the focus.
  • The evaluation of appearance quality is done using metrics that are not directly comparing individual images but rather statistical differences between sets of images.
  • Future Work:
    • The authors suggest that further improvements could be achieved by adding decoder self-attention, which could leverage temporal relations within the sequence even more effectively.
  • The development of a detector for generated gait transfer videos is presented as a first step towards addressing the potential misuse of such technology, implying a need for continued research in this area.

Summary updated

BibTeX
@inproceedings{mahpod2023ctrgan,
  author = {Shahar Mahpod and Noam Gaash and Hay Hoffman and Gil Ben-Artzi},
  title = {{CTrGAN: Cycle Transformers GAN for Gait Transfer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure