TL;DR
This paper introduces FusePose, a novel framework for 3D human pose estimation that effectively fuses Inertial Measurement Unit (IMU) data with multi-view camera images. By leveraging a parametric human kinematic model and performing fusion in a kinematic space, FusePose significantly improves accuracy, especially in challenging scenarios with occlusions, and ensures anatomically plausible poses.
What Problem Does This Paper Address?
Estimating the 3D pose of a human from images is a fundamental problem in computer vision, with applications ranging from action recognition to human-computer interaction. However, a major challenge is the inherent ambiguity and unreliability introduced by occlusions and self-occlusions in visual data. While Inertial Measurement Units (IMUs) offer a complementary, occlusion-free source of motion information, effectively fusing this heterogeneous data with visual information has been a persistent challenge. Previous attempts often fail to fully utilize the rich information from either IMUs or vision, or struggle with the heterogeneity of the data.
Methods
FusePose tackles this problem by introducing a framework that operates within a parametric human kinematic model. The core idea is to represent human poses not just as 3D joint coordinates, but as a set of pose parameters that describe the configuration of a skeletal model. This allows for a more consistent and anatomically plausible representation.
The paper proposes three fusion approaches:
- NaiveFuse: A baseline approach that fuses simplified IMU data (bone vectors) with estimated 3D poses in Euclidean space. It uses a thresholding mechanism to decide when to replace vision-derived bone vectors with IMU-derived ones, primarily for hard cases with occlusions.
- KineFuse: This method moves the fusion into the kinematic space. It aligns calibrated IMU raw data (as local rotations) with pose parameters derived from vision data using an inverse kinematics (IK) layer. This approach leverages the kinematic model to ensure bone length consistency and incorporate twist rotations, which are lost in Euclidean space fusion.
- AdaDeepFuse: The most advanced approach, AdaDeepFuse, is an end-to-end trainable deep learning framework. It adaptively fuses pose parameters from both IMU and vision data using a learned "AdaFuse" module. This allows the network to dynamically determine the contribution of each sensor modality, leading to more robust and accurate pose estimation.
A key technical contribution is the implementation of canonical IK and forward kinematics (FK) layers within a deep neural network. These layers are parameter-free and differentiable, enabling seamless integration into deep learning pipelines and facilitating fusion in kinematic space.
Datasets
The proposed framework is evaluated on two datasets:
- Total Capture: This dataset is a large-scale benchmark that includes IMUs, multi-view videos, and 3D human pose ground truth. It's used for training and evaluating the proposed methods.
- Human3.6M: This dataset is used to evaluate the generalization capabilities of the framework. Since Human3.6M does not contain real IMU data, simulated IMU information is generated from the ground truth pose annotations.
Key Insights and Ideas
- Kinematic Space Fusion is Crucial: Moving the fusion process from Euclidean space to kinematic space is key to ensuring global skeleton consistency and incorporating local rotation information (like bone twists) that is lost in simpler fusion methods.
- Parametric Models Enable Better Fusion: Representing poses using parametric models (pose parameters) rather than just 3D joint coordinates provides a more structured and anatomically constrained representation, which is better suited for integrating sensor data.
- Adaptive Fusion Outperforms Static Approaches: An end-to-end trainable adaptive fusion module (AdaFuse) can dynamically weigh the contributions of IMU and vision data, leading to superior performance compared to fixed fusion strategies.
- IMUs Help with Occlusions: IMU data, being occlusion-free, directly addresses the primary weakness of vision-based pose estimation, significantly improving accuracy in challenging scenarios.
- IK/FK Layers are Efficient and Effective: Implementing IK and FK as differentiable layers within a neural network is an efficient way to leverage kinematic constraints without significant computational overhead.
Key Results
The FusePose framework demonstrates significant improvements over baseline methods and state-of-the-art approaches on the Total Capture dataset:
- NaiveFuse improves pose estimation by 5.4% over the vision-only baseline.
- KineFuse achieves a 10.0% improvement over the baseline and surpasses previous state-of-the-art methods that use IMUs only for testing by 8.6%.
- AdaDeepFuse achieves the best performance, with a 13.1% improvement over the baseline and outperforms state-of-the-art methods that use IMUs for both training and testing by 8.5%.
- Qualitative results show that FusePose can effectively handle self-occlusions, producing more plausible and anatomically correct poses compared to vision-only methods.
- Experiments on Human3.6M demonstrate the generalization capability of NaiveFuse and KineFuse.
Limitations and Future Work
- Reliance on Known Bone Lengths: The current framework relies on a predefined rest T-pose with known bone lengths. The authors acknowledge that obtaining this in real-world applications can be challenging and propose integrating bone length prediction into the framework in the future.
- Temporal Continuity of IMUs: While the current work focuses on fusing spatial information, the temporal continuity and stability of IMU data, especially during fast motion, is a valuable aspect that could be further explored in future work.
- Upstream Joint Prediction: Failure cases often stem from unsatisfactory predictions of upstream joints, which then propagate errors to downstream joints. Improving the robustness of upstream joint estimation remains an area for future research.
Summary updated