Using joint angles based on the international biomechanical standards for human action recognition and related tasks

Hao Ni, Kevin Schlegel, Lei Jiang, Hao Ni 0001arXiv 2024 · DOI · arXiv

Introduction

Human Action Recognition (HAR) has traditionally relied on "skeleton data"—sequences of 3D coordinates representing joint positions. While effective, this representation suffers from viewpoint dependency, subject-specific variations (like height), and a lack of semantic interpretability. This paper proposes a shift toward standardized biomechanical joint angles based on International Society of Biomechanics (ISB) standards. By representing motion through viewpoint-independent, human-interpretable angles, the authors aim to create models that are more robust and efficient.

The Case for Biomechanical Standards

The biomechanics community has utilized rigorous standards for describing human motion since the 1980s. In contrast, the computer vision community often uses raw XYZ coordinates or simple dot-product angles. The authors argue that adopting Joint Coordinate Systems (JCS) can address several fundamental issues:

  • Viewpoint Independence: Coordinates change with camera movement, but joint angles relative to the body do not.
  • Subject Normalization: Differences in limb length are naturally factored out when focusing on rotation.
  • Interpretability: Experts like clinicians or coaches describe movement in terms of flexion, extension, or abduction, not Cartesian coordinates.

Methodology: From Keypoints to Joint Coordinate Systems

The core contribution is a systematic conversion of keypoint data into JCS. Following ISB recommendations, local coordinate systems are defined for each joint, decomposing motion into three primary planes: sagittal (flexion/extension), frontal (abduction/adduction), and transverse (internal/external rotation).

Standardized local coordinate systems applied to a human skeleton model to define joint angles.

Unlike simpler angle representations, these biomechanical angles provide a nearly bijective mapping. Given known bone lengths, the original skeleton can be perfectly reconstructed from the angles. To ensure performance gains were due to the data representation rather than specific architectures, the authors tested the approach using standard 1D Convolutions, LSTMs, and Transformers.

Results and Discussion

A key finding was the "Hard" Task Advantage. On the Skeletics "Hard" subset, joint angles notably outperformed raw keypoints, capturing the subtle nuances required to distinguish between similar actions, such as different types of ball throws.

Performance comparison (mAP vs. Accuracy) on the Skeletics (Hard) dataset, highlighting how angle-based representations (circles) often cluster higher than raw skeleton data (crosses/pluses).

Robustness to Rotation

In stress tests involving random input rotations, keypoint-based models saw accuracy drop significantly (e.g., from 82.9% to 47.3%). In contrast, joint-angle models maintained perfectly stable performance, as the representation is mathematically rotation-invariant by design.

Data Efficiency and Ensembling

Joint angles represent motion using approximately 1/3 of the data required for 3D keypoints. Furthermore, the best overall results were achieved by ensembling keypoint-based and angle-based models. On NTU RGB+D, an ensemble reached 84.4% accuracy, outperforming individual keypoint (82.9%) and angle (70.6%) models.

Conclusion and Future Work

While joint angles show immense promise for interpretability and robustness, they exhibit sensitivity to noise in low-quality data (like Kinect). Future work includes optimizing architectures like Graph Convolutional Networks (GCNs) for joint angle structures and expanding datasets to include the additional keypoints required for full ISB standard implementation. The authors have released a Python package to facilitate the conversion of keypoint data into these standardized biomechanical angles, bridging the gap between computer vision and biomechanics.

Summary updated

Abstract

Keypoint data has received a considerable amount of attention in machine learning for tasks like action detection and recognition. However, human experts in movement such as doctors, physiotherapists, sports scientists and coaches use a notion of joint angles standardised by the International Society of Biomechanics to precisely and efficiently communicate static body poses and movements. In this paper, we introduce the basic biomechanical notions and show how they can be used to convert common keypoint data into joint angles that uniquely describe the given pose and have various desirable mathematical properties, such as independence of both the camera viewpoint and the person performing the action. We experimentally demonstrate that the joint angle representation of keypoint data is suitable for machine learning applications and can in some cases bring an immediate performance gain. The use of joint angles as a human meaningful representation of kinematic data is in particular promising for applications where interpretability and dialog with human experts is important, such as many sports and medical applications. To facilitate further research in this direction, we will release a python package to convert keypoint data into joint angles as outlined in this paper.
BibTeX
@misc{ni2024using,
  author = {Hao Ni and Kevin Schlegel and Lei Jiang and Hao Ni},
  title = {{Using joint angles based on the international biomechanical standards for human action recognition and related tasks}},
  year = {2024},
  eprint = {2406.17443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17443},
}
Figure