Introduction
Automated Action Quality Assessment (AQA) in figure skating is a complex task that requires balancing technical precision with artistic interpretation. Human judges evaluate performances using two distinct metrics: the Technical Element Score (TES), based on the execution of specific elements like jumps and spins, and the Program Component Score (PCS), which assesses the overall artistic performance and its synchronization with music.
Current AI models often face three significant hurdles: modality mismatch (fusing audio and video indiscriminately), lack of granularity (predicting a single total score instead of individual element scores), and the computational cost of processing long-range temporal dependencies in routines lasting several minutes.
Methodology
To address these challenges, we propose the Two-Stream Mamba Pyramid Network. This architecture explicitly separates the evaluation into two pathways: a Visual Context Stream for TES and a Visual-Audio Context Stream for PCS, mirroring the actual judging criteria of the sport.
Mamba Pyramid and Down Sampling
At the core of our network are Mamba Down Sampling (MDS) blocks. Unlike standard Transformers, which have quadratic computational complexity relative to sequence length, Mamba utilizes state-space models (SSMs) to achieve linear-time complexity. This allows the model to efficiently process long competition videos while maintaining a global receptive field. By stacking these blocks into a pyramid structure, the network captures features at multiple temporal resolutions, accommodating the varying durations of different skating elements, from quick jumps to extended step sequences.
Multi-level Cross Attention Fusion (MCAF)
For the artistic assessment (PCS), the model employs a Multi-level Cross Attention Fusion module. This allows the audio features to "query" the visual stream across the pyramid levels, ensuring the final score reflects the rhythmic and emotional alignment between the skater's movements and the music.
Datasets and Evaluation
The model was primarily trained and evaluated on the FineFS dataset, which contains 1,604 high-resolution videos with fine-grained annotations for action categories, temporal boundaries, and individual element scores. To test the model's robustness and transferability, we also conducted evaluations on the Fis-V and FS1000 benchmarks.
Results
Our approach achieves state-of-the-art performance across multiple metrics. On the FineFS benchmark, the model reached a Spearman correlation of 0.80 for TES and 0.96 for PCS in Free Skating routines, significantly outperforming previous leaders like LUSD-Net.
Ablation studies highlighted several key findings:
- Modality Alignment: Including audio in the TES stream actually decreased performance, confirming that technical elements are best judged visually. Conversely, audio was critical for the high accuracy of PCS.
- Efficiency: The Mamba-based encoder outperformed CNN-based alternatives while using significantly fewer parameters (40.45M vs. 64.24M), demonstrating its suitability for long-form sports video analysis.
Conclusion
By aligning neural architecture with the fundamental rules of figure skating judging, the Two-Stream Mamba Pyramid Network provides a more accurate and explainable framework for automated sports assessment. The use of Mamba blocks effectively solves the long-range dependency problem, offering a scalable and efficient solution for high-precision action quality assessment.
Summary updated