D^2-STX: Decoupling Spatial-Temporal Cross-Attention for Dual-Branch Repetitive Action Counting

Xiaoai Wang, Hang Wang, Yan Liu, Huan Hu, Bruce X.b. YuCVPR 2026 Findings 2026 · OpenReview

TL;DR

$D^2$-STX introduces a dual-branch framework for Repetitive Action Counting (RAC) that combines RGB video data with human pose streams. By explicitly decoupling spatial alignment from temporal synchronization using a novel cross-attention mechanism, the model achieves state-of-the-art counting accuracy while remaining robust to background clutter, occlusions, and viewpoint changes.

What Problem Does This Paper Address?

Repetitive Action Counting (RAC) aims to determine how many times a person performs a periodic action (like push-ups or jumping jacks) in a video. This task is challenging because it must be action-agnostic—working across diverse, unseen activities and environments.

Most existing methods rely solely on RGB video features. However, appearance-based models easily get confused by cluttered backgrounds, lighting changes, or body-part occlusions. Conversely, relying purely on skeletal pose data provides structural stability but discards crucial contextual and fine-grained motion details.

Fusing these two modalities seems like an obvious fix, but standard Transformer-based cross-attention strategies struggle with heterogeneous data structures. Typically, they flatten and concatenate video and pose tokens into a single sequence. This causes spatial-temporal coupling, where the attention mechanism conflates cross-frame temporal dependencies with within-frame spatial relationships, leading to ambiguous attention patterns and degraded performance.

Figure 1. Our dual-branch motivation. Video stream is sensitive to background clutter, body-part occlusion, and viewpoint changes, whereas pose stream remain structurally stable and invariant to these factors but lack fine-grained appearance and motion detail.

Methods

The authors propose $D^2$-STX (Decoupled Dual-branch Spatial-Temporal Cross-Attention), which processes video and pose data in parallel and fuses them hierarchically without mixing up space and time.

  1. Dual-Branch Representation: The video stream uses standard backbones (like Swin Transformer or RX3D101) to extract appearance features. The pose stream uses BlazePose to extract 2D coordinates for 25 keypoints, which are tokenized via a pre-trained CTR-GCN.
  2. Frame-wise Spatial Cross-Attention (FSCA): Instead of flattening everything, FSCA operates independently on each frame. For example, in the pose-guided mode, pose features act as queries to actively retrieve relevant visual details from the dense video features, aligning the two modalities semantically in space.
  3. Temporal Cross-Attention: After spatial alignment, the model applies either Joint-wise Temporal Cross-Attention (JTCA) for pose-guided features or Pixel-wise Temporal Cross-Attention (PTCA) for video-guided features. This isolates the temporal dimension, enforcing modality-consistent motion dynamics over time.
  4. Bidirectional Fusion: The framework can simultaneously run both pose-guided and video-guided branches, combining their predicted action density maps using a learnable dynamic weighting parameter ($\alpha$).
Illustration of our proposed D^2-STX framework.

Datasets

The model was evaluated on three public RAC benchmarks:

  • RepCount-A: Contains 1,041 videos with fine-grained annotations of the start and end moments of each action cycle.
  • UCFRep: An extension of the UCF101 dataset consisting of 526 videos in a relatively controlled setting.
  • Countix: A large-scale, challenging dataset of 4,588 training and 2,719 testing real-world YouTube videos featuring significant scene diversity and speed fluctuations.

Key Insights and Ideas

  • The Power of Decoupling: Separating spatial cross-attention from temporal cross-attention prevents the semantic degradation common in standard multi-modal Transformers. It ensures that the model synchronizes when an action happens only after aligning where the body parts are.
  • Information Asymmetry Exploitation: Video data is dense but noisy; pose data is sparse but structurally clean. Using pose joints as queries to probe dense video features allows the model to actively recover from keypoint detection errors by pulling in local visual context.
  • Cross-Dataset Robustness: Because the structural pose stream is invariant to environmental changes, the fused model generalizes remarkably well to entirely unseen datasets without any fine-tuning.

Key Results

$D^2$-STX consistently outperformed previous non-large-model state-of-the-art approaches across all benchmarks, measured primarily by Mean Absolute Error (MAE) and Off-By-One accuracy (OBO):

  • RepCount-A: Achieved an MAE reduction of 4.3% and boosted OBO accuracy to 0.500 (a +0.097 improvement over the previous best method).
  • UCFRep & Countix: Reduced MAE by 1.1% and 1.5%, respectively, demonstrating strong performance in both controlled and unconstrained real-world environments.
  • Generalization: When trained on RepCount-A and tested directly on UCFRep, $D^2$-STX outperformed the previous best method by 6.3% in MAE and 12.6% in OBO.
Figure 3. Density plot visualization of action repetition count predictions. The ground-truth (GT) label, the video branch count prediction, and our D2-STX’s count prediction are shown for action repetitions of two “pull-up” videos.

Limitations and Future Work

The authors note a few key limitations:

  • 2D Pose Constraints: The model relies on 2D pose estimation (BlazePose) and a GCN pre-trained on a different domain (NTU-RGB+D), which introduces a domain gap and limits performance if the initial 2D keypoint tracking fails severely.
  • The LLM Shadow: While $D^2$-STX beats traditional vision models, recent massive Large Language Model (LLM) approaches like CountLLM achieve slightly higher absolute accuracy. However, $D^2$-STX remains vastly more practical for real-world deployment, boasting an inference speed of 10–20ms on a single GPU compared to the massive compute footprint required by LLMs.

Future work will focus on integrating higher-quality 3D human pose data and exploring lightweight Large Multi-modal Models (LMMs) to build more explainable and cost-effective counting systems.

Summary updated

Abstract

Repetitive Action Counting (RAC) aims to estimate the number of category-agnostic periodic actions in videos. However, most existing methods exclusively rely on video modality features, making them susceptible to complex background interference and inadequate modeling of human motion structure. To address these challenges, we present D2-STX, a framework for RAC using a dual-branch decoupled spatial-temporal cross-attention mechanism, which efficiently fuses complementary structured and appearance-based motion representations from video and pose data streams, respectively. Our D2-STX achieves effective fine-grained fusion at frame, joint, and pixel levels, enhancing the spatial-temporal distinguishability of both video and pose streams. Extensive experiments on three public RAC datasets (i.e., RepCount-A, UCFRep, and Countix) demonstrate that our method consistently outperforms state-of-the-art approaches across all benchmarks. Further ablations and visualizations validate the superiority of the proposed key designs in our method, providing new insights for video understanding and multi-modal spatial-temporal fusion.
BibTeX
@inproceedings{wang2026d,
  author = {Xiaoai Wang and Hang Wang and Yan Liu and Huan Hu and Bruce X.B. Yu},
  title = {{D^2-STX: Decoupling Spatial-Temporal Cross-Attention for Dual-Branch Repetitive Action Counting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=ufpKzVfJ0j},
}
Figure