Expert Comment Generation Considering Sports Skill Level Using a Large Multimodal Model with Video and Spatial-Temporal Motion Features

Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Ren TogoSensors 2025 · DOI

TL;DR

This paper introduces a novel method for generating personalized, skill-level-aware expert comments for athletes by combining video analysis with spatial-temporal motion features. By leveraging a Large Multimodal Model (LMM) and a Spatial-Temporal Attention Graph Convolutional Network (STA-GCN), the system provides actionable feedback tailored to an athlete's specific skill level and movements, moving beyond generic advice.

What Problem Does This Paper Address?

Personalized feedback is crucial for athletic skill development, but obtaining expert coaching is often limited by resource constraints. Existing automated systems primarily focus on skill evaluation through video analysis, often overlooking the nuanced spatial-temporal dynamics of human movement. Furthermore, when expert comments are generated, they typically assume a predetermined skill level and fail to incorporate detailed motion information. This leaves a gap in providing truly actionable, individualized advice that accounts for both the athlete's current proficiency and the fine-grained details of their performance.

Methods

The proposed method integrates three key components:

  1. Visual Feature Extraction: A pre-trained visual encoder (languagebind) processes the input sports video to extract visual features, which are then transformed into token embeddings.
  2. Skill Level Classification and Motion Feature Extraction: A Spatial-Temporal Attention Graph Convolutional Network (STA-GCN) is employed. This network models human motion as a graph, where joints are nodes and bones are edges, to capture spatial-temporal dynamics. The STA-GCN extracts motion features and then classifies the athlete's skill level based on these features.
  3. Large Multimodal Model (LMM) for Comment Generation: The extracted visual tokens, motion features, and the classified skill level are fed into a pre-trained LMM. This multimodal input, along with a predefined prompt, enables the LMM to generate detailed, context-specific, and actionable expert comments.

The novelty lies in explicitly incorporating the classified skill level and detailed spatial-temporal motion features as inputs to the LMM, ensuring that the generated feedback is not only relevant to the visual performance but also tailored to the athlete's proficiency and specific movements.

Datasets

The experiments were conducted using basketball videos from the Ego-Exo4D dataset. This dataset provided videos, skill levels, and expert comments. For pre-training the STA-GCN, 154 basketball motion data points with four skill levels from the Ego-Exo4D dataset were used. The LMM utilized was GPT-40. The Ego-Exo4D dataset comprised 1313 samples, which were randomly split into 90% for training and 10% for testing.

Key Insights and Ideas

  • Skill Level is Paramount: Explicitly incorporating the athlete's skill level into the feedback generation process significantly improves the relevance and nuance of the generated comments. Without it, comments can be generic or misaligned with the athlete's actual needs.
  • Motion Matters (A Lot): Spatial-temporal motion features are critical for understanding the dynamics of athletic performance. Simply looking at visuals isn't enough; capturing how joints move over time and space provides the detailed information needed for precise advice.
  • Multimodality is Key: Combining visual information, motion data, and skill level classification into a single model allows for a holistic understanding of performance, leading to more comprehensive and actionable feedback.
  • LMMs as Powerful Commentators: Large Multimodal Models, when properly conditioned with rich, multimodal inputs, can generate human-like, expert-level commentary that goes beyond simple descriptions.

Key Results

The proposed method (PM) achieved the highest scores across all evaluation metrics (BLEU-4, METEOR, ROUGE-L) in quantitative evaluations, outperforming various comparative methods. For instance, the PM achieved a BLEU-4 score of 1.75 x 10⁻², METEOR of 0.256, and ROUGE-L of 0.156. Ablation studies demonstrated that incorporating skill level information, motion features, and visual tokenization all individually contributed to improved performance, with their combined effect being the most significant. Qualitative evaluations showed that the PM's comments were more specific, actionable, and closely aligned with ground truth expert comments compared to other methods, including general video captioning models and a prior expert comment generation method (ExpertAF).

Limitations and Future Work

  • Limitations:
    • The paper notes that the original expert comments in the dataset sometimes lacked specificity regarding which body parts were involved, requiring a preprocessing step to clarify this.
  • While the method effectively identifies areas for improvement, it may not explicitly mention every detail present in the ground truth, though it offers complementary advice.
  • Future Work:
    • The authors suggest that their method effectively addresses the limitations of existing approaches by providing precise and actionable feedback tailored to individual performance. The study successfully meets research objectives by improving expert comment generation through a model that accounts for skill level and motion features, considering spatial-temporal information.

Summary updated

Abstract

In sports training, personalized skill assessment and feedback are crucial for athletes to master complex movements and improve performance. However, existing research on skill transfer predominantly focuses on skill evaluation through video analysis, addressing only a single facet of the multifaceted process required for skill acquisition. Furthermore, in the limited studies that generate expert comments, the learner's skill level is predetermined, and the spatial-temporal information of human movement is often overlooked. To address this issue, we propose a novel approach to generate skill-level-aware expert comments by leveraging a Large Multimodal Model (LMM) and spatial-temporal motion features. Our method employs a Spatial-Temporal Attention Graph Convolutional Network (STA-GCN) to extract motion features that encapsulate the spatial-temporal dynamics of human movement. The STA-GCN classifies skill levels based on these motion features. The classified skill levels, along with the extracted motion features (intermediate features from the STA-GCN) and the original sports video, are then fed into the LMM. This integration enables the generation of detailed, context-specific expert comments that offer actionable insights for performance improvement. Our contributions are twofold: (1) We incorporate skill level classification results as inputs to the LMM, ensuring that feedback is appropriately tailored to the learner's skill level; and (2) We integrate motion features that capture spatial-temporal information into the LMM, enhancing its ability to generate feedback based on the learner's specific actions. Experimental results demonstrate that the proposed method effectively generates expert comments, overcoming the limitations of existing methods and offering valuable guidance for athletes across various skill levels.
BibTeX
@inproceedings{seino2025expert,
  author = {Tatsuki Seino and Naoki Saito and Takahiro Ogawa and Satoshi Asamizu and Ren Togo},
  title = {{Expert Comment Generation Considering Sports Skill Level Using a Large Multimodal Model with Video and Spatial-Temporal Motion Features}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25020447},
}
Figure