Introduction
Repetitive action counting—the process of quantifying cycles in movements like pull-ups, squats, or industrial tasks—is a cornerstone of modern fitness tracking and physical therapy. Traditionally, this task has been approached using "video-level" methods that process raw RGB video data. However, these methods are computationally intensive and often distracted by "noise" such as background lighting or clothing changes.
This paper introduces PoseRAC (Pose Saliency Transformer for Repetitive Action Counting), a framework that shifts the paradigm from pixel-heavy video processing to lightweight pose-level analysis. By focusing on the "essence" of human motion through key joint coordinates, PoseRAC achieves superior accuracy with a fraction of the computational cost.
Methodology
PoseRAC redefines repetitive counting by identifying Pose Saliency Representation (PSR)—the two most distinct poses that define a movement cycle (e.g., the "arms-hanging" and "arms-pull-up" positions in a pull-up). The pipeline consists of four stages:
- Pose Estimation: Utilizing off-the-shelf models like BlazePose, the system extracts 2D/3D joint coordinates, reducing data volume by approximately 80%.
- Transformer Encoder: A Transformer architecture embeds these keypoints into high-level features. The training employs Metric Learning (Triplet Margin Loss) to ensure that similar poses are clustered together while distinct action phases are pushed apart in the feature space.
- Pose Mapping: This module classifies each frame's pose into an action class based on its saliency rather than arbitrary start/end timestamps.
- Action-trigger Module: A lightweight logic gate that increments the count only when the two salient poses of a specific action are detected in the correct sequence.
Datasets and Annotation
To support this pose-level approach, the authors introduced Pose Saliency Annotation (PSA). Unlike traditional datasets that mark action boundaries, PSA identifies the specific indices of salient poses. The researchers released two re-annotated benchmarks:
- RepCount-pose: 487 training videos across 8 action classes.
- UCFRep-pose: A subset of UCFRep re-annotated for 5 action classes.
Key Results
PoseRAC demonstrates a significant leap over previous state-of-the-art (SOTA) models like TransRAC and RepNet. On the RepCount-pose test set, PoseRAC achieved an Off-By-One (OBO) metric of 0.56, nearly doubling the performance of previous video-level methods.
Efficiency is a primary advantage: the model processes frames at 20ms per frame, making it roughly 10x faster than traditional architectures. Furthermore, its lightweight nature allows it to be trained on a standard CPU in under two hours, a task typically requiring high-end GPUs for video-level transformers.
Conclusion
PoseRAC proves that saliency is more effective than redundancy in action counting. By stripping away environmental noise and focusing on key human poses, the model achieves state-of-the-art results with extreme efficiency. While its performance is dependent on the quality of the underlying pose estimation, it sets a new benchmark for real-time, robust action counting in diverse environments.
Summary updated