Abstract
We propose a view-invariant method towards the assessment of the quality of human movements which does not rely on skeleton data. Our end-to-end convolutional neural network consists of two stages, where at first a view-invariant trajectory descriptor for each body joint is generated from RGB images, and then the collection of trajectories for all joints are processed by an adapted, pre-trained 2D CNN (e.g. VGG-19 or ResNeXt-50) to learn the relationship amongst the different body parts and deliver a score for the movement quality. We release the only publicly-available, multi-view, non-skeleton, non-mocap, rehabilitation movement dataset (QMAR), and provide results for both cross-subject and cross-view scenarios on this dataset. We show that VI-Net achieves average rank correlation of 0.66 on cross-subject and 0.65 on unseen views when trained on only two views. We also evaluate the proposed method on the single-view rehabilitation dataset KIMORE and obtain 0.66 rank correlation against a baseline of 0.62.
Introduction
Objective assessment of human movement quality is critical in clinical rehabilitation for conditions such as Parkinson’s disease or stroke recovery. While traditional methods rely on subjective clinical observation or expensive 3D motion capture (Mocap) and depth sensors, computer vision offers a scalable alternative. However, standard RGB-based systems often suffer from viewpoint dependency, where performance drops significantly when the camera angle changes from the training perspective.
VI-Net addresses this challenge by providing a view-invariant assessment of movement quality from standard RGB video. By transforming 2D joint trajectories into a normalized, canonical representation, the system maintains high scoring accuracy even from previously unseen camera angles, without requiring specialized hardware.
Methodology: The VI-Net Architecture
The VI-Net architecture is an end-to-end neural network composed of two primary stages: the View-invariant Trajectory Descriptor Module (VTDM) and the Movement Score Module (MSM).
View-invariant Trajectory Descriptor Module (VTDM)
The process begins by extracting 2D heatmaps for 15 key body joints from video frames. These heatmaps are stacked temporally to create a "trajectory map" for each joint. To achieve view-invariance, the VTDM integrates a Spatial Transformer Network (STN). The STN learns to "unwarp" these 2D trajectories—compensating for rotation, scale, and translation—to map them into a view-independent space. This approach treats 2D trajectories as affine transformations of a base movement, allowing the model to normalize perspective automatically.
Movement Score Module (MSM)
The normalized trajectory maps are fed into a modified pre-trained 2D CNN (such as VGG-19 or ResNeXt-50). This module learns the spatial and temporal relationships between different joint trajectories—such as the coordination between arm swing and leg stride—to output a final quality score. By utilizing 2D convolutions on temporal stacks rather than heavy 3D convolutions, the model remains computationally efficient while capturing essential movement dynamics.
Datasets and Evaluation
The researchers evaluated VI-Net using two primary datasets focusing on rehabilitation movements.
- QMAR (Quality of Movement Assessment for Rehabilitation): A new multi-view dataset introduced in this work. It features 38 subjects performing walking and sit-to-stand actions across six camera views, including simulated gait abnormalities scored by severity.
- KIMORE: A publicly available single-view dataset of 78 subjects performing lower-back exercises, used to demonstrate the model's versatility across different movement types.
Key Results
VI-Net demonstrates robust performance across both cross-view and cross-subject scenarios:
- Cross-View Generalization: When trained on only two viewpoints and tested on unseen angles in the QMAR dataset, VI-Net achieved a 0.65 rank correlation (Spearman’s rho).
- Cross-Subject Performance: The model achieved a 0.66 rank correlation when scoring individuals not seen during training.
- State-of-the-Art Comparison: On the KIMORE dataset, VI-Net reached a 0.66 correlation, significantly outperforming the 0.62 baseline established by traditional 3D temporal methods.
Conclusion
VI-Net represents a significant advancement in accessible clinical assessment by enabling view-invariant movement scoring from standard RGB video. While the model effectively handles perspective changes, future work will address challenges related to long-term occlusions and validate the system with real-world patient data in unconstrained environments.
Summary updated