Likert Scoring with Grade Decoupling for Long-term Action Assessment

Angchi Xu, Ling-An Zeng, Wei-Shi ZhengCVPR 2022 · OpenReview

Abstract

Long-term action quality assessment (AQA) is the task of evaluating how well an action is performed by estimating a quality score from a long video. Intuitively, long-term actions involve segments exhibiting different levels of skill, which we refer to as performance grades. For example, technical highlights and faults may appear in the same routine. To explore this latent relationship, we design a novel Likert scoring paradigm inspired by psychometrics. We quantify performance grades explicitly and generate a final quality score by combining quantitative values with corresponding responses estimated from the video. Our model, the Grade-decoupling Likert Transformer (GDLT), extracts grade-specific features through a Transformer decoder architecture with learnable queries. GDLT achieves state-of-the-art results on two major long-term action assessment datasets.

Introduction

Action Quality Assessment (AQA) aims to determine how well an action was performed. While short actions like diving are relatively straightforward, long-term actions—such as a 3-minute figure skating routine—are complex. A single long video often contains a "mixed bag" of performance levels: a skater might execute a perfect triple axel (Excellent) but later stumble on a footwork sequence (Poor).

Traditional methods typically use a "black box" approach, employing a neural network to directly regress a single number from the entire video. This ignores the inherent multi-level nature of human performance and makes it difficult for models to learn the nuances that distinguish a "good" performance from a "great" one. To address this, we propose a paradigm shift toward Likert scoring.

Overview of the Likert scoring paradigm, illustrating how video frames are decoupled into performance grades to calculate a final score.

The GDLT Framework

The Grade-decoupling Likert Transformer (GDLT) mimics the Likert scale used in psychology, where respondents rate agreement on a scale (e.g., "Strongly Disagree" to "Strongly Agree"). The architecture consists of four main components:

  1. Feature Extraction: Video segments are processed using a backbone (e.g., Video Swin Transformer) to extract initial representations.
  2. Temporal Context Encoder (TCE): This module captures long-range dependencies across the entire performance, ensuring the model understands the global context of the action.
  3. Grade-aware Decoder (GAD): This is the core innovation. The model maintains a set of learnable "grade prototypes" (queries). Through cross-attention, these prototypes extract grade-aware features from the video, effectively identifying where "Excellent" or "Poor" work occurs.
  4. Likert Scoring Module (LSM): This module performs quantification (assigning fixed values to grades) and response estimation (predicting the intensity of each grade). The final score is a weighted sum of these values.
The complete GDLT architecture, showing the flow from feature extraction through context enhancement and grade decoupling to the final Likert scoring.

To prevent the model from predicting identical features for every grade, we employ a Diversity Loss (triplet loss) to force the grade-aware features to remain distinct.

Experimental Evaluation

We evaluated GDLT on two primary long-term AQA benchmarks using Spearman’s rank correlation coefficient (SRCC):

  • Rhythmic Gymnastics (RG): Comprising 1,000 videos across four apparatus types. GDLT achieved an average SRCC of 0.765, significantly outperforming previous benchmarks like ACTION-NET (0.728).
  • Figure Skating (Fis-V): Comprising 500 videos of ladies' singles programs. GDLT achieved an average SRCC of 0.761, surpassing MS-LSTM and other prior methods.

Ablation studies revealed that K=4 grades (Excellent, Good, Fair, Poor) is the optimal configuration. Fewer grades lack sufficient granularity, while more grades introduce ambiguity that degrades performance.

Key Insights and Conclusion

The Likert scoring paradigm acts as a "bridge" that simplifies the mapping from complex high-dimensional video data to a single scalar score. Beyond performance gains, this method offers improved interpretability. Visualizations demonstrate that the "Poor" grade prototype's attention spikes during athlete errors (e.g., falls), while the "Excellent" prototype focuses on technical highlights.

While the model currently relies on fixed quantification intervals and high-quality video backbones, it represents a significant step toward more transparent and accurate automated action assessment. Future work may explore learnable grade widths and applications in less structured domains like daily living tasks.

Summary updated

Abstract

Long-term action quality assessment is a task of evaluating how well an action is performed, namely, estimating a quality score from a long video. Intuitively, long-term actions generally involve parts exhibiting different levels of skill, and we call the levels of skill as performance grades. For example, technical highlights and faults may appear in the same long-term action. Hence, the final score should be determined by the comprehensive effect of different grades exhibited in the video. To explore this latent relationship, we design a novel Likert scoring paradigm in-spired by the Likert scale in psychometrics, in which we quantify the grades explicitly and generate the final quality score by combining the quantitative values and the corresponding responses estimated from the video, instead of performing direct regression. Moreover, we extract grade-specific features, which will be used to estimate the responses of each grade, through a Transformer decoder architecture with diverse learnable queries. The whole model is named as Grade-decoupling Likert Transformer (GDLT), and we achieve state-of-the-art results on two long-term action assessment datasets. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Project page https://isee-ai.cn/-angchi/CVPR22_GDLT.html
BibTeX
@inproceedings{xu2022likert,
  author = {Angchi Xu and Ling-An Zeng and Wei-Shi Zheng},
  title = {{Likert Scoring with Grade Decoupling for Long-term Action Assessment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=4jVjgNcudg},
}

Loading related content...

Figure