TL;DR
This paper introduces Trampoline-AQA, the first large-scale, dual-modality (RGB + optical flow) dataset for assessing the quality of long-duration, high-speed actions in trampoline gymnastics. They also propose a novel framework combining a Temporal Feature Enhancer (TFE) and a causal cross-modal attention module (FCCA) to achieve state-of-the-art performance in this challenging domain.
What Problem Does This Paper Address?
Action Quality Assessment (AQA) is crucial for sports analysis, athlete training, and judging. However, existing research and datasets primarily focus on short, isolated actions. Sports like trampoline gymnastics involve complex, continuous routines lasting 25-35 seconds with intricate aerial maneuvers. Current methods struggle to model these long-term dependencies, fine-grained temporal dynamics, and the subtle causal relationships between actions, leading to inaccurate and less robust scoring, especially when dealing with motion ambiguities. This paper aims to bridge this gap by providing a more suitable dataset and a more sophisticated modeling approach.
Methods
The authors propose a dual-stream framework that leverages both RGB video and optical flow data.
- Optical Flow Extraction: They use the RAFT model for optical flow extraction and introduce an Optical Flow Anomaly Detection (OFA) module. OFA uses a triple-verification mechanism (color distribution, intensity, and temporal continuity) to filter out noisy or unreliable optical flow maps, ensuring higher quality dynamic information.
- Feature Extraction: Two established backbone models, I3D for RGB streams and X3D for optical flow, are used to extract spatiotemporal features.
- Temporal Feature Enhancer (TFE): This novel module enhances local action features by strengthening temporal representations. It uses sequence-adaptive kernels and attention weights to dynamically adjust features, improving the capture of local dynamic patterns.
- Forward-looking Causal Cross-Modal Attention (FCCA): This core module performs causal and cross-modal fusion. It uses historical and future masks to model causal relationships between actions, allowing the model to understand how early errors propagate. It also facilitates cross-modal attention between RGB and optical flow features, leading to more discriminative embeddings.
- Score Prediction: A score distribution autoencoder is employed to model the probability distribution of scores, rather than just predicting a single value, leading to more stable and accurate predictions.
Datasets
- Trampoline-AQA (New Dataset): This is the primary contribution. It comprises 206 video clips from major trampoline gymnastics competitions (2018-2024), averaging 30.05 seconds in length. It features dual-modality data (RGB and optical flow) and rich annotations including competition type, score labels (difficulty, execution, flight time, etc.), and action counts. This dataset addresses the limitations of existing AQA datasets which are typically much shorter.
- UNLV-Dive Dataset: Used for evaluating the generalization capabilities of their proposed framework. This dataset contains videos of men's 10m platform diving.
Key Insights and Ideas
- Long-duration AQA is fundamentally different: The paper highlights that short-term action modeling is insufficient for sports like trampoline gymnastics, where the entire routine's quality depends on the interplay of many continuous actions.
- Optical flow is crucial for dynamic sports: While RGB captures appearance, optical flow is essential for understanding the precise motion dynamics, especially in high-speed, complex maneuvers. However, raw optical flow can be noisy, necessitating robust filtering.
- Causality matters in action sequences: Understanding how an early mistake (e.g., a slight wobble during takeoff) impacts the rest of the routine is key to accurate AQA. The FCCA module's causal modeling explicitly addresses this.
- Dual-modality fusion needs careful design: Simply concatenating features isn't enough. The FCCA module's cross-modal attention allows modalities to inform each other, leading to a richer representation.
- Modeling score distributions improves robustness: Predicting a distribution rather than a single score helps capture uncertainty and leads to more stable predictions, especially in subjective domains like sports judging.
Key Results
- State-of-the-art performance: The proposed method achieves a Spearman correlation (SRC) of 0.938 on the Trampoline-AQA dataset, significantly outperforming all baseline methods.
- Strong generalization: The framework also achieves an SRC of 0.882 on the UNLV-Dive dataset, demonstrating its effectiveness beyond trampoline gymnastics.
- Module contributions: Ablation studies show that the OFA module significantly improves optical flow quality, TFE enhances temporal feature modeling, and FCA is crucial for causal reasoning and cross-modal fusion. Combining OFA and FCA yields substantial gains.
- Efficiency: The OFA module processes an average 31-second video in about 45 seconds, indicating reasonable efficiency for real-time applications.
Limitations and Future Work
- Scalability and Cost: The reliance on expert annotations for the Trampoline-AQA dataset is costly. The computational expense of RAFT for optical flow extraction also limits real-time practicality.
- Future Work: The authors plan to reduce annotation dependence through semi-supervised learning and synthetic labeling. They also aim to enhance the time efficiency of the end-to-end system. Additionally, they plan to embed later temporal attention mechanisms into FCCA to further improve error propagation modeling and overall robustness.
Summary updated