Hybrid Dynamic-static Context-aware Attention Network for Action Assessment in Long Videos

Ling-An Zeng, Fa-Ting Hong, Wei-Shi Zheng, Qi-Zhi Yu, Wei Zeng, Yaowei Wang, Jian-Huang Lai

Introduction

Automatic Action Quality Assessment (AQA) seeks to develop computer vision systems capable of scoring athletic performances with the precision of human judges. Despite advancements, traditional models often struggle with two primary challenges: the omission of static cues and the inherent noise in long-form video. In sports like rhythmic gymnastics or figure skating, a single static posture—such as the specific extension of a limb—can be as critical as the motion itself. Furthermore, in routines lasting several minutes, treating every segment with equal importance leads to "score dilution," where critical errors or highlights are obscured by mundane transitions.

To address these hurdles, we introduce ACTION-NET, a novel framework that combines motion data with static athlete postures through a context-aware attention mechanism to provide more accurate scoring in long videos.

The ACTION-NET Framework

ACTION-NET utilizes a two-stream architecture designed to capture the full spectrum of an athlete's performance:

  • Dynamic Stream: Employs an I3D (Inflated 3D ConvNet) backbone to extract motion features from video segments, capturing the temporal flow of the routine.
  • Static Stream: Focuses specifically on the athlete's form. By using YOLOv3 for person detection and ResNet for feature extraction from the resulting crops, the model isolates the athlete from background distractions, ensuring that technical posture is accurately represented.
The ACTION-NET architecture, featuring parallel dynamic and static streams integrated via a context-aware attention module.

Context-aware Attention Mechanism

The core innovation of ACTION-NET is its Context-aware Attention Module, which acts as the system's evaluative "brain." It consists of two specialized units:

  1. Temporal Clipwise GCN Unit (TCG-U): A Graph Convolutional Network that models the relationships between disparate time segments. This allows the model to understand the global context of a specific movement relative to the entire performance.
  2. Attention Unit (ATT-U): This unit assigns weights to individual segments. High weights are prioritized for technical maneuvers or visible mistakes, while lower weights are assigned to irrelevant parts like setup or transitions.

Experimental Evaluation

The model was evaluated on two benchmarks: the established MIT-Skating dataset (150 videos) and a newly introduced Rhythmic Gymnastics dataset. The latter provides a more rigorous test for long-video assessment, containing 1,000 high-quality videos across four categories (ball, clubs, hoop, and ribbon), annotated with official international difficulty and execution scores.

Performance was measured using Spearman’s rank correlation (ρ), which quantifies the alignment between model predictions and expert human scores.

Results and Analysis

ACTION-NET demonstrated superior performance across both datasets:

  • MIT-Skating: Achieved a ρ of 0.615, surpassing the previous state-of-the-art (0.590).
  • Rhythmic Gymnastics: Set a high baseline, notably reaching a ρ of 0.708 in the "Hoop" category.
Visualization of attention weights assigned to different frames, highlighting the model's ability to distinguish critical technical moments from background transitions.

Ablation studies confirmed that the hybrid approach is essential. Models using only dynamic or only static features consistently underperformed compared to the integrated ACTION-NET. Furthermore, the inclusion of human detection in the static stream proved vital; without it, background noise significantly degraded scoring accuracy.

Conclusion

ACTION-NET effectively bridges the gap between motion-centric models and the nuanced requirements of sports judging. By integrating static posture analysis and context-aware attention, the framework identifies the most salient moments in long routines. Future work will explore the integration of fine-grained pose estimation (skeleton data) to further refine the assessment of technical form and reduce the computational intensity of the two-stream GCN operations.

Summary updated

BibTeX
@inproceedings{zeng2020hybrid,
  author = {Ling-An Zeng and Fa-Ting Hong and Wei-Shi Zheng and Qi-Zhi Yu and Wei Zeng and Yaowei Wang and Jian-Huang Lai},
  title = {{Hybrid Dynamic-static Context-aware Attention Network for Action Assessment in Long Videos}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=JRFHj66lOw9},
}
Figure