TL;DR
This paper introduces VI-Net, a novel view-invariant deep learning method for assessing human movement quality directly from RGB videos, without relying on 3D skeleton data. They also release QMAR, a new multi-view dataset for rehabilitation movement assessment, and demonstrate VI-Net's effectiveness across cross-subject, cross-view, and single-view scenarios.
What Problem Does This Paper Address?
Assessing the quality of human movement is crucial in various fields, including sports, skill evaluation, and especially patient rehabilitation. Clinicians often rely on subjective assessments to gauge functional mobility. Automating this process using computer vision could lead to more objective, cost-effective, and accessible evaluations. However, existing methods often struggle with view-invariance, meaning their performance degrades significantly when presented with viewpoints different from those they were trained on. Many approaches attempt to overcome this by using 3D pose data from depth sensors (like Kinect) or motion capture systems. While effective, these methods have limitations: Kinect performance is sensitive to environmental conditions, and motion capture is expensive and impractical for real-world settings. Furthermore, recent deep learning methods for 3D pose estimation from RGB still face challenges with view-invariance, often requiring multiple views for training. This paper addresses the critical need for a view-invariant method that works directly with RGB data and doesn't require explicit 3D pose information or multiple training views.
Methods
The proposed method, VI-Net, is an end-to-end convolutional neural network composed of two main modules:
- View-Invariant Trajectory Descriptor Module (VTDM):
- Joint Heatmap Generation: The process begins by extracting 2D joint heatmaps from RGB video frames using OpenPose. The authors select the first 15 joints from the BODY-25 version of OpenPose, arguing that the remaining joints offer repetitive information.
- Trajectory Descriptor: For each body joint, its heatmaps across a sequence of frames are stacked to form a 3D heatmap volume. A 2D convolution filter is then applied to aggregate spatial poses over time, generating a "trajectory descriptor map" for each joint.
- View-Invariance Forging: To achieve view-invariance, a Spatial Transformer Network (STN) is applied to these trajectory descriptor maps. The STN learns to estimate an affine transformation matrix that maps the descriptor to a canonical view, effectively making it invariant to translation, rotation, and scale.
- Movement Score Module (MSM):
- Feature Aggregation: The view-invariant trajectory descriptors from all body joints are stacked to form a global descriptor.
- Quality Scoring: This global descriptor is fed into an adapted, pre-trained 2D convolutional neural network (CNN), such as VGG-19 or ResNeXt-50. These networks are fine-tuned to learn the relationships between joint trajectories and predict a movement quality score. The authors note that using pre-trained models allows them to leverage learned features from natural images.
The entire VI-Net is trained end-to-end. For classification of discrete quality scores, a cross-entropy loss is used.
Datasets
- QMAR (Quality of Movement Assessment for Rehabilitation): This is a newly introduced dataset by the authors. It's a multi-view dataset featuring 38 healthy subjects performing normal movements and simulated Parkinson's and Stroke ailments. The movements include walking and sit-to-stand actions, recorded from 6 different camera views. The dataset includes RGB, depth, and skeleton data, though only RGB is used in this work. QMAR aims to provide a benchmark for view-invariant movement assessment.
- KIMORE: A recently released single-view rehabilitation dataset. It contains RGB images and score annotations for five types of lower-back exercises performed by healthy individuals and patients with Parkinson's, Stroke, and back pain. This dataset is used for single-view evaluation.
Key Insights and Ideas
- Repurposing Temporal Pose Information: The core idea is that while appearance changes drastically with viewpoint, the underlying spatio-temporal trajectories of body joints are affine transformations of each other. By focusing on these trajectories and making them view-invariant, movement quality can be assessed regardless of the camera angle.
- View-Invariance Without 3D Pose: A significant contribution is achieving view-invariance without relying on explicit 3D pose estimation or depth sensors, making the method more practical for real-world applications.
- STN's Role in View-Invariance: The Spatial Transformer Network is crucial for achieving view-invariance by learning to normalize the orientation and scale of the joint trajectory descriptors.
- Flexibility of the MSM: The use of adapted pre-trained CNNs (VGG-19, ResNeXt-50) in the MSM module allows for flexibility and leverages existing powerful feature extractors.
- Dataset Contribution: The QMAR dataset fills a critical gap by providing a multi-view RGB dataset specifically for rehabilitation movement quality assessment, enabling more robust evaluation of view-invariant methods.
Key Results
- Cross-Subject Performance: VI-Net achieved competitive results on the QMAR dataset, outperforming C3D and I3D baselines with an average rank correlation of 0.66 when using all views for training.
- Cross-View Performance (Single-View Training): When trained on a single view and tested on unseen views, VI-Net achieved an average rank correlation of 0.65. Performance was better for walking movements (up to 0.73) than for sit-to-stand movements, which are more prone to long-term occlusions.
- Cross-View Performance (Two-View Training): Training with two views significantly improved cross-view performance, reaching up to 0.92 for walking movements.
- Single-View Performance (KIMORE): On the KIMORE dataset, VI-Net achieved an average rank correlation of 0.66, outperforming C3D and I3D baselines.
- STN Impact: The ablation study showed that while STN's impact was less pronounced when multiple views were used for training (as the MSM could learn from diverse orientations), it was crucial for achieving view-invariance when trained on a single view.
Limitations and Future Work
- Occlusion Sensitivity: VI-Net's performance degrades in scenarios with long-term occlusions, partly due to limitations in OpenPose's ability to consistently produce heatmaps in such cases.
- Movement Type Specific Training: The current VI-Net requires separate training for each movement type.
- Future Work:
- Investigate 3D pose estimation methods to generate more robust heatmaps less affected by occlusions.
- Develop multitask learning to recognize movement type and score simultaneously.
- Explore unsupervised training of view-invariant features from existing multi-view datasets for transfer learning.
Summary updated