TL;DR
This paper tackles the challenge of accurately counting repetitive actions in videos, especially when backgrounds change abruptly. They introduce a novel two-branch framework that combines RGB and motion features, using a motion attention module and a temporal self-similarity matrix reconstruction loss to improve robustness. A key innovation is a dynamic loss weighting mechanism that adapts to video variance, leading to state-of-the-art performance.
What Problem Does This Paper Address?
Counting repetitive actions in videos, like sit-ups or push-ups, is a crucial task with applications in fitness tracking and physical education. The main hurdle is accurately identifying the periodic pattern of these actions. Existing methods often rely solely on RGB features from individual frames. However, this approach falters when the background changes significantly between frames, even if the action itself is the same. These abrupt background changes can create feature discrepancies, leading to errors in counting. This paper aims to overcome this limitation by explicitly learning and leveraging motion features, which are more invariant to background changes, to enhance the accuracy of repetitive action counting.
Methods
The proposed method employs a two-branch framework:
- RGB Branch: This branch uses a standard transformer-based pipeline for repetitive action counting, similar to prior work. It extracts features from the RGB frames and uses a Temporal Self-Similarity Module (TSM) to capture temporal relationships.
- Motion Branch: This is the core novelty. It introduces a Motion Attention Module (MAM) that fuses optical flow information with RGB features. The MAM uses flow features as queries and RGB features as keys/values to implicitly highlight motion features within the RGB data. This helps in learning motion patterns that are robust to background variations.
To further improve robustness and temporal correspondence, the paper introduces:
- Temporal Self-Similarity Matrix Reconstruction Loss: This loss function aims to enhance the similarity between frames depicting the same action moment. It works by reconstructing a temporal self-similarity matrix, effectively reinforcing connections between frames with similar motion patterns and downplaying noise from transient background elements. This is applied to both RGB and motion branches.
- Variance-Integrated Loss Weights Generation: Instead of fixed weights, this technique dynamically adjusts the importance of the RGB and motion branches during training. It calculates the variance of each video and assigns higher weights to the motion branch when the variance (indicating background changes) is high. This adaptive approach ensures that the model prioritizes the most relevant features for each specific video.
Datasets
The paper uses two widely adopted benchmarks for training and evaluation:
- RepCount dataset [11]: This dataset consists of two parts, Part A and Part B. The authors primarily use Part A, which contains diverse action types.
- UCFRep dataset [42]: This dataset is derived from UCF101 and contains videos of 23 different repetitive action classes.
The paper also performs cross-dataset generalization experiments, training on RepCount and testing on UCFRep, to demonstrate the robustness of their approach.
Key Insights and Ideas
- Motion is Key for Robustness: The paper strongly emphasizes that motion features, derived from optical flow, are crucial for handling background variations in repetitive action counting. Relying solely on RGB features is insufficient when backgrounds change drastically.
- Attention for Motion Feature Enhancement: The Motion Attention Module (MAM) is a clever way to guide the model to focus on relevant motion cues by using flow as a "query" to attend to RGB features.
- Temporal Correspondence as a Regularizer: The temporal self-similarity matrix reconstruction loss acts as a powerful regularizer, forcing the model to learn consistent representations of actions across frames, thereby reducing the impact of spurious background movements.
- Dynamic Loss Weighting for Adaptability: The variance-integrated loss weighting is a significant contribution. It allows the model to intelligently balance the contributions of the RGB and motion branches based on the inherent difficulty (variance) of each video, leading to more effective collaborative learning.
Key Results
The proposed method achieves state-of-the-art performance on both the RepCount and UCFRep datasets.
- On the RepCount dataset, the method surpasses the previous state-of-the-art TransRAC [11] by 5.90% in MAE and 9.47% in OBO.
- On the UCFRep dataset, it outperforms TransRAC [11] by 5.3% in MAE and Context [42] by 3% in OBO.
- In cross-dataset generalization (training on RepCount, testing on UCFRep), the method achieves a significant improvement of 11.74% in MAE and 2.8% in OBO compared to previous methods.
- Ablation studies confirm the effectiveness of each proposed component: the MAM, the matrix reconstruction loss, and the variance-integrated loss weights.
Limitations and Future Work
- Multiple Repetitive Objects: The paper acknowledges that counting repetitive actions with multiple distinct repetitive objects in a single video is a future direction.
- Weakly Supervised/Self-Supervised Learning: The authors suggest exploring weakly supervised or self-supervised methods for repetitive action counting.
- Multimodal Fusion: Combining other modalities, such as language descriptions, with video data is also proposed as a promising avenue for future research.
Summary updated