Action Assessment by Joint Relation Graphs

Jiahui Pan, Jibin Gao, Wei-Shi ZhengICCV 2019 · OpenReview

Abstract We present a new model to assess the performance of actions from videos, through graph-based joint relation modelling. Previous works mainly focused on the whole scene including the performer's body and background, yet they ignored the detailed joint interactions. This is insufficient for fine-grained, accurate action assessment, because the action quality of each joint is dependent of its neighbouring joints. Therefore, we propose to learn the detailed joint motion based on the joint relations. We build trainable Joint Relation Graphs, and analyze joint motion on them. We propose two novel modules, the Joint Commonality Module and the Joint Difference Module, for joint motion learning. The Joint Commonality Module models the general motion for certain body parts, and the Joint Difference Module models the motion differences within body parts. We evaluate our method on six public Olympic actions for performance assessment. Our method outperforms previous approaches (+0.0912) and the whole-scene analysis (+0.0623) in the Spearman's Rank Correlation. We also demonstrate our model's ability to interpret the action assessment process.

Introduction

Action Quality Assessment (AQA) is a challenging task that requires distinguishing between high-level athletic performances. Traditional approaches often rely on "whole scene" analysis, which can be influenced by irrelevant background information, or track individual joints in isolation, missing the critical nuances of human movement.

The core challenge in AQA is capturing fine-grained coordination. For instance, in gymnastics, the position of a knee is only "correct" in the context of the hip and ankle's simultaneous movement. This paper introduces a novel graph-based approach that models these intricate relationships. By decomposing performance into the general kinetics of body parts (Commonality) and the coordination between neighboring joints (Difference), the proposed model provides a more accurate and interpretable assessment.

Conceptual framework for assessing action performance based on body part kinetics and joint coordination.

Methodology

The proposed framework utilizes Joint Relation Graphs to analyze motion. The process begins with feature extraction using a standard I3D (Inflated 3D ConvNet) on both the whole scene and local patches cropped around detected joints.

The overall architecture of the proposed Action Assessment model, featuring the Joint Commonality and Joint Difference modules.

The technical core consists of two learnable graphs:

  1. Spatial Relation Graph ($A_s$): Models how joints interact within a single time step.
  2. Temporal Relation Graph ($A_p$): Models how a joint’s position at time $t-1$ influences its neighbors at time $t$.

These graphs feed into two specialized modules:

  • Joint Commonality Module: Uses graph convolutions to aggregate motion across groups of joints (e.g., an entire limb) to understand general body part kinetics.
  • Joint Difference Module: Acts as a "coordination engine" by calculating motion differences between a joint and its neighbors. This captures the "signature" of synchronized movement.

Finally, a regression module fuses these features with whole-scene information to predict a final performance score.

Experiments and Results

The model was evaluated on two benchmarks:

  • AQA-7 Dataset: 1,106 videos of Olympic actions like Diving, Gymnastic Vault, and Skiing.
  • JIGSAWS Dataset: Surgical activity assessment involving robotic tools performing tasks like suturing.

On the AQA-7 dataset, the model achieved a Spearman’s Rank Correlation of 0.7849, significantly outperforming the previous state-of-the-art (0.6937). It also showed superior performance in surgical tasks, demonstrating the versatility of the joint relation logic. Notably, the model remained robust even when using different pose estimation tools (Mask-RCNN vs. AlphaPose).

Discussion and Insights

A key insight of this work is modeling coordination as the difference between neighboring joints. This allows the model to capture synchronized movement patterns more effectively than absolute positions. Furthermore, the learnable nature of the graphs allows the model to automatically identify which joints are most critical for a specific action—for example, focusing on the hips and shoulders for gymnastics. This provides a level of interpretability missing from "black box" models, potentially identifying specific technical flaws in a performance.

Conclusion

By focusing on the relational dynamics of human joints rather than just global motion, this approach sets a new standard for fine-grained action assessment. While the model currently depends on accurate pose estimation and carries higher computational costs than whole-frame methods, its ability to provide detailed, interpretable feedback opens doors for applications in automated coaching and surgical training.

Summary updated

Abstract

We present a new model to assess the performance of actions from videos, through graph-based joint relation modelling. Previous works mainly focused on the whole scene including the performer's body and background, yet they ignored the detailed joint interactions. This is insufficient for fine-grained, accurate action assessment, because the action quality of each joint is dependent of its neighbouring joints. Therefore, we propose to learn the detailed joint motion based on the joint relations. We build trainable Joint Relation Graphs, and analyze joint motion on them. We propose two novel modules, the Joint Commonality Module and the Joint Difference Module, for joint motion learning. The Joint Commonality Module models the general motion for certain body parts, and the Joint Difference Module models the motion differences within body parts. We evaluate our method on six public Olympic actions for performance assessment. Our method outperforms previous approaches (+0.0912) and the whole-scene analysis (+0.0623) in the Spearman's Rank Correlation. We also demonstrate our model's ability to interpret the action assessment process.
BibTeX
@inproceedings{pan2019action,
  author = {Jiahui Pan and Jibin Gao and Wei-Shi Zheng},
  title = {{Action Assessment by Joint Relation Graphs}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=HQxs6u7k2O},
}
Figure