PoseRAC: Pose Saliency Transformer for Repetitive Action Counting

Ziyu Yao, Xuxin Cheng, Yuexian ZouarXiv 2023 · DOI · arXiv

Introduction

Repetitive action counting—the process of quantifying cycles in movements like pull-ups, squats, or industrial tasks—is a cornerstone of modern fitness tracking and physical therapy. Traditionally, this task has been approached using "video-level" methods that process raw RGB video data. However, these methods are computationally intensive and often distracted by "noise" such as background lighting or clothing changes.

This paper introduces PoseRAC (Pose Saliency Transformer for Repetitive Action Counting), a framework that shifts the paradigm from pixel-heavy video processing to lightweight pose-level analysis. By focusing on the "essence" of human motion through key joint coordinates, PoseRAC achieves superior accuracy with a fraction of the computational cost.

Methodology

PoseRAC redefines repetitive counting by identifying Pose Saliency Representation (PSR)—the two most distinct poses that define a movement cycle (e.g., the "arms-hanging" and "arms-pull-up" positions in a pull-up). The pipeline consists of four stages:

  1. Pose Estimation: Utilizing off-the-shelf models like BlazePose, the system extracts 2D/3D joint coordinates, reducing data volume by approximately 80%.
  2. Transformer Encoder: A Transformer architecture embeds these keypoints into high-level features. The training employs Metric Learning (Triplet Margin Loss) to ensure that similar poses are clustered together while distinct action phases are pushed apart in the feature space.
  3. Pose Mapping: This module classifies each frame's pose into an action class based on its saliency rather than arbitrary start/end timestamps.
  4. Action-trigger Module: A lightweight logic gate that increments the count only when the two salient poses of a specific action are detected in the correct sequence.
Comparison between common video-level methods and the proposed pose-level PoseRAC framework.

Datasets and Annotation

To support this pose-level approach, the authors introduced Pose Saliency Annotation (PSA). Unlike traditional datasets that mark action boundaries, PSA identifies the specific indices of salient poses. The researchers released two re-annotated benchmarks:

  • RepCount-pose: 487 training videos across 8 action classes.
  • UCFRep-pose: A subset of UCFRep re-annotated for 5 action classes.

Key Results

PoseRAC demonstrates a significant leap over previous state-of-the-art (SOTA) models like TransRAC and RepNet. On the RepCount-pose test set, PoseRAC achieved an Off-By-One (OBO) metric of 0.56, nearly doubling the performance of previous video-level methods.

Efficiency is a primary advantage: the model processes frames at 20ms per frame, making it roughly 10x faster than traditional architectures. Furthermore, its lightweight nature allows it to be trained on a standard CPU in under two hours, a task typically requiring high-end GPUs for video-level transformers.

Performance comparison of PoseRAC against state-of-the-art methods in terms of OBO metric and inference speed.

Conclusion

PoseRAC proves that saliency is more effective than redundancy in action counting. By stripping away environmental noise and focusing on key human poses, the model achieves state-of-the-art results with extreme efficiency. While its performance is dependent on the quality of the underlying pose estimation, it sets a new benchmark for real-time, robust action counting in diverse environments.

Summary updated

Abstract

This paper presents a significant contribution to the field of repetitive action counting through the introduction of a new approach called Pose Saliency Representation. The proposed method efficiently represents each action using only two salient poses instead of redundant frames, which significantly reduces the computational cost while improving the performance. Moreover, we introduce a pose-level method, PoseRAC, which is based on this representation and achieves state-of-the-art performance on two new version datasets by using Pose Saliency Annotation to annotate salient poses for training. Our lightweight model is highly efficient, requiring only 20 minutes for training on a GPU, and infers nearly 10x faster compared to previous methods. In addition, our approach achieves a substantial improvement over the previous state-of-the-art TransRAC, achieving an OBO metric of 0.56 compared to 0.29 of TransRAC. The code and new dataset are available at https://github.com/MiracleDance/PoseRAC for further research and experimentation, making our proposed approach highly accessible to the research community.
BibTeX
@misc{yao2023poserac,
  author = {Ziyu Yao and Xuxin Cheng and Yuexian Zou},
  title = {{PoseRAC: Pose Saliency Transformer for Repetitive Action Counting}},
  year = {2023},
  eprint = {2303.08450},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.08450},
}
Figure