Abstract
Long-term action quality assessment (AQA) is the task of evaluating how well an action is performed by estimating a quality score from a long video. Intuitively, long-term actions involve segments exhibiting different levels of skill, which we refer to as performance grades. For example, technical highlights and faults may appear in the same routine. To explore this latent relationship, we design a novel Likert scoring paradigm inspired by psychometrics. We quantify performance grades explicitly and generate a final quality score by combining quantitative values with corresponding responses estimated from the video. Our model, the Grade-decoupling Likert Transformer (GDLT), extracts grade-specific features through a Transformer decoder architecture with learnable queries. GDLT achieves state-of-the-art results on two major long-term action assessment datasets.
Introduction
Action Quality Assessment (AQA) aims to determine how well an action was performed. While short actions like diving are relatively straightforward, long-term actions—such as a 3-minute figure skating routine—are complex. A single long video often contains a "mixed bag" of performance levels: a skater might execute a perfect triple axel (Excellent) but later stumble on a footwork sequence (Poor).
Traditional methods typically use a "black box" approach, employing a neural network to directly regress a single number from the entire video. This ignores the inherent multi-level nature of human performance and makes it difficult for models to learn the nuances that distinguish a "good" performance from a "great" one. To address this, we propose a paradigm shift toward Likert scoring.
The GDLT Framework
The Grade-decoupling Likert Transformer (GDLT) mimics the Likert scale used in psychology, where respondents rate agreement on a scale (e.g., "Strongly Disagree" to "Strongly Agree"). The architecture consists of four main components:
- Feature Extraction: Video segments are processed using a backbone (e.g., Video Swin Transformer) to extract initial representations.
- Temporal Context Encoder (TCE): This module captures long-range dependencies across the entire performance, ensuring the model understands the global context of the action.
- Grade-aware Decoder (GAD): This is the core innovation. The model maintains a set of learnable "grade prototypes" (queries). Through cross-attention, these prototypes extract grade-aware features from the video, effectively identifying where "Excellent" or "Poor" work occurs.
- Likert Scoring Module (LSM): This module performs quantification (assigning fixed values to grades) and response estimation (predicting the intensity of each grade). The final score is a weighted sum of these values.
To prevent the model from predicting identical features for every grade, we employ a Diversity Loss (triplet loss) to force the grade-aware features to remain distinct.
Experimental Evaluation
We evaluated GDLT on two primary long-term AQA benchmarks using Spearman’s rank correlation coefficient (SRCC):
- Rhythmic Gymnastics (RG): Comprising 1,000 videos across four apparatus types. GDLT achieved an average SRCC of 0.765, significantly outperforming previous benchmarks like ACTION-NET (0.728).
- Figure Skating (Fis-V): Comprising 500 videos of ladies' singles programs. GDLT achieved an average SRCC of 0.761, surpassing MS-LSTM and other prior methods.
Ablation studies revealed that K=4 grades (Excellent, Good, Fair, Poor) is the optimal configuration. Fewer grades lack sufficient granularity, while more grades introduce ambiguity that degrades performance.
Key Insights and Conclusion
The Likert scoring paradigm acts as a "bridge" that simplifies the mapping from complex high-dimensional video data to a single scalar score. Beyond performance gains, this method offers improved interpretability. Visualizations demonstrate that the "Poor" grade prototype's attention spikes during athlete errors (e.g., falls), while the "Excellent" prototype focuses on technical highlights.
While the model currently relies on fixed quantification intervals and high-quality video backbones, it represents a significant step toward more transparent and accurate automated action assessment. Future work may explore learnable grade widths and applications in less structured domains like daily living tasks.
Summary updated