Rubric-Constrained Figure Skating Scoring

Arushi Rai, Adriana KovashkaWACV 2025 · DOI

TL;DR

This paper introduces a novel method for automatically scoring figure skating performances by leveraging the sport's official rubrics. By decoupling technical element difficulty from quality and using natural language descriptions of criteria, the approach achieves interpretable scoring and improved precision, offering athletes actionable feedback for practice.

What Problem Does This Paper Address?

Automatic scoring of complex, multi-action sports like figure skating presents a significant challenge. Traditional action quality assessment (AQA) methods often treat the entire video as a single unit, making it difficult to pinpoint which specific actions contributed to the overall score. This lack of interpretability is a major drawback for athletes seeking to improve their performance. Furthermore, existing AQA models tend to entangle the difficulty and quality of an element, with difficulty often dominating the score due to its magnitude. This is problematic because quality (how well an element is executed) is what athletes can directly influence. The paper aims to bridge this gap by developing a system that scores each element individually, focuses on quality, and provides explanations rooted in the sport's official scoring rubrics, all without requiring explicit element segmentation annotations.

Methods

The proposed method, "Rubric-Constrained Figure Skating Scoring," introduces several key innovations:

  • Element Transformer: A transformer encoder-decoder architecture is used to process video clips. The decoder employs learnable "element queries" that implicitly localize and generate embeddings for each figure skating element. This is inspired by object detection transformers (DETR) but adapted for temporal segmentation without explicit annotations.
  • Implicit Segmentation Regularization (ISR): To encourage element queries to focus on relevant video segments rather than background transitions, two regularization techniques are applied to the cross-attention maps:
  • Orthogonality Loss: Minimizes the dot product between cross-attention distributions of different queries, promoting distinct attention patterns for each element.
  • Peak Loss: Encourages attention distributions to be concentrated around a single contiguous clip, modeled as a normal distribution centered on the clip with the highest attention value.
  • Rubric-Constrained Scoring: Instead of a traditional regression head, the method uses the element embeddings and natural language descriptions of rubric items (obtained from the ISU scoring protocol) to compute the Grade of Execution (GOE). A pretrained text encoder (CLIP) generates embeddings for rubric text. The cosine similarity between element embeddings and rubric text embeddings is used to weigh the contribution of each rubric item to the final GOE score.
  • Action Masking: To ensure that only relevant rubric items are considered for a given element (e.g., jump-related criteria for a jump), an action masking mechanism is employed.
  • Pretraining: To address the challenge of aligning visual element embeddings with textual rubric descriptions, two pretraining strategies are used:
  • Visual-Text Pretraining (PTvt): A contrastive triplet loss aligns high-scoring elements with positive rubric items and low-scoring elements with negative ones.
  • Visual Pretraining (PTvis): Another triplet loss enforces geometric structure in the visual embedding space, ensuring similar elements are closer together. These pretraining steps are jointly optimized.

Datasets

The primary dataset used for training and evaluation is Fis-V [24], which contains approximately 400 training and 100 testing videos of figure skating performances. The paper also mentions using FS1000 [21] in supplementary materials. For evaluating the implicit segmentation aspect, a small subset of Fis-V was manually annotated with element segmentation.

Key Insights and Ideas

  • Rubrics as a Source of Interpretability: The core idea of using official sport rubrics as the basis for scoring is a powerful way to make automated systems interpretable. This moves beyond black-box models and directly links scoring to established criteria.
  • Implicit Segmentation is Sufficient: The paper demonstrates that explicit, annotated element segmentation is not strictly necessary for achieving good performance. By using regularization techniques, the model can implicitly learn to focus on relevant video segments for each element.
  • Decoupling Difficulty and Quality is Crucial: By separating the base value (difficulty) from the grade of execution (quality), the model can focus on the aspect that athletes can most directly improve, making the feedback more actionable.
  • Vision-Language Models for Rubric Alignment: Leveraging models like CLIP to embed rubric text and align it with visual features is an effective way to bridge the gap between visual performance and textual criteria.

Key Results

  • Improved Scoring Precision: The proposed method significantly reduces the Mean Squared Error (MSE) for Technical Element Score (TES) prediction, achieving an MSE of 9.34 on the Fis-V benchmark, compared to state-of-the-art methods with MSEs around 19.05-19.53.
  • Enhanced Ranking Capability: The method also improves Spearman rank correlation, reaching 0.84, indicating a better ability to rank performances correctly.
  • Effective Implicit Segmentation: The implicit segmentation regularization (ISR) leads to a substantial increase in element queries attending to actual element segments (from 12.5% to 35.7% in one experiment).
  • Ablation Studies: Simplification of rubric text and the use of pretraining (especially joint pretraining) significantly boost performance. Action masking also improves scoring precision.

Limitations and Future Work

  • Order of Elements: While the method improves the ability of queries to attend to elements, it does not guarantee that elements are attended to in the correct temporal order. The authors suggest that explicit element supervision could address this.
  • Generalizability: The method is currently tailored to figure skating. Adapting it to other sports with different rubric structures or less structured performances might be challenging.
  • Future Directions: The authors suggest exploring further peak loss and ordering loss techniques. They also propose that adding explicit element supervision could lead to improved performance and explainability, potentially enabling direct use by athletes. Investigating the use of more advanced vision-language models and exploring the application to other sports are also potential avenues.

Summary updated

BibTeX
@inproceedings{rai2025rubric,
  author = {Arushi Rai and Adriana Kovashka},
  title = {{Rubric-Constrained Figure Skating Scoring}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00882},
}
Figure