TL;DR
This paper introduces "UJ-AQA-CricketVision," a novel dataset for cricket action quality assessment (AQA), and a new I3D-AE-LSTM model that combines spatio-temporal and pose keypoint information. This approach achieves state-of-the-art performance, outperforming existing methods by effectively analyzing complex, short-duration sports actions.
What Problem Does This Paper Address?
Action Quality Assessment (AQA) aims to automatically evaluate the quality of human actions from video. This is a challenging task, especially in sports like cricket, where subtle variations in technique can significantly impact performance. Existing AQA methods often treat actions as single representations (either spatio-temporal or pose-based), potentially missing crucial nuances. Furthermore, many AQA datasets are limited in size or scope, making it difficult to develop robust models for complex actions. The paper addresses these gaps by introducing a new, detailed cricket dataset and a model that leverages both visual and skeletal information for a more comprehensive analysis.
Methods
The core of the proposed method is a 2-Stream Autoencoder (AE) model, named I3D-AE-LSTM. This model takes a two-pronged approach:
- Spatio-temporal Stream: This stream uses an Inflated 3D (I3D) Convolutional Neural Network (CNN) to extract features from video frames. The I3D features are then fed into an autoencoder for dimensionality reduction and feature learning.
- Pose Keypoint Stream: This stream utilizes ViTPose to extract skeletal keypoints from each frame. These keypoints are then processed by a separate autoencoder, which uses Long Short-Term Memory (LSTM) layers to capture temporal dependencies in the pose data.
These two streams are then fused using a weighted combination formula: $F_{fused} = \alpha F_v + \beta F_p$. The fused features are then passed through an LSTM-based network for feature aggregation and finally a regression model (specifically, a Multi-Layer Perceptron or MLP) to predict the action quality score. The model is designed to learn from individual body part scores, offering a more granular analysis than methods that only consider an overall score.
Datasets
- UJ-AQA-CricketVision (New Dataset): This is the primary contribution. It comprises 8,540 video clips of cricket batsmen performing various strokes. Each clip is annotated with detailed phase breakdowns (buildup, execution, follow-through) and individual body part scores (head, shoulder, hands, hips, feet). The dataset was collected from YouTube footage and annotated using the VIA tool by two annotators, with a scoring guideline developed in conjunction with cricket experts.
- Existing Datasets (for benchmarking): The paper mentions using UNLV Olympic, MTL-AQA, and AQA-7 datasets as comparative benchmarks. Specifically, the MTL-AQA dataset was adapted for benchmarking the proposed model.
Key Insights and Ideas
- Synergy of Spatio-temporal and Pose Data: The most significant insight is that combining visual frame information with skeletal pose data leads to a substantial improvement in AQA performance, especially for complex and rapid actions. This suggests that neither modality alone is sufficient for a complete understanding.
- Body Part Specific Scoring: Analyzing and scoring individual body parts (head, shoulders, etc.) provides a more nuanced understanding of technique than relying solely on an overall action score. This allows for more targeted feedback and a deeper analysis of what makes an action "good" or "bad."
- Dataset Design Matters for AQA: The UJ-AQA-CricketVision dataset's detailed, multi-phase, and body-part-specific annotations are crucial for developing models that can capture the subtle dynamics of sports actions. The dataset's focus on short, complex sequences addresses a limitation in existing benchmarks.
- Autoencoders for Feature Representation: The use of autoencoders in both streams is key to learning compact and informative feature representations from both raw frames and pose keypoints, enabling efficient processing and dimensionality reduction.
Key Results
- The proposed I3D-AE-LSTM model achieved a Spearman Rank Correlation (SRC) score of 0.84 on the UJ-AQA-CricketVision dataset.
- This performance outperforms existing state-of-the-art methods on this dataset.
- An ablation study showed that integrating SlowFast with ViTPose significantly boosted performance, achieving an SRC of 0.80 in Experiment 3, Trial 3.
- The model demonstrated strong performance across individual body parts, with SRC scores ranging from 0.83844 (feet) to 0.84146 (head).
- When benchmarked on the adapted MTL-AQA dataset, the I3D-AE-LSTM model achieved a promising SRC of 0.97.
Limitations and Future Work
- Data Scarcity for Low Scores: The model tends to overestimate actual values, particularly for action scores between 0-2. This is attributed to the significantly fewer samples available in the dataset for these lower-scored actions.
- Focus on Execution Phase: The initial study focused primarily on the execution phase for AQA. Future work will involve analyzing all phases (buildup, execution, follow-through) to understand their combined impact.
- Exploring Other Dataset Aspects: Future research will also explore the impact of other factors like dismissals and different stroke types on action scores.
- Improving Low-Score Prediction: The authors suggest enriching the dataset with more samples representing lower-scored actions to improve the model's accuracy and correlation for these categories.
- Action Score Smoothing: Applying a consistent action score across all frames in a sequence significantly enhances learning efficacy, a technique that could be further explored.
Summary updated