Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective

Feiyu Yao, Zongkai Wu, Li YiAAAI 2024 · DOI

Abstract

Reconstructing full-body 3D human motion from sparse tracking data—typically just a VR headset and two controllers—remains a significant challenge in AR/VR. This paper introduces a novel framework that models the human skeleton as a Body Pose Graph (BPG) and treats motion reconstruction as a missing node prediction problem. By integrating multi-modal features, employing a temporal pyramid for motion continuity, and utilizing expressive edges (static, dynamic, and latent), the proposed method achieves state-of-the-art accuracy. Specifically, it reduces the Mean Per Joint Position Error (MPJPE) to 3.34 cm on the AMASS dataset, with a notable 18-22% improvement in lower-body joint prediction compared to previous methods.

Introduction

In Augmented and Virtual Reality (AR/VR), creating realistic avatars that mirror user movement is essential for immersion. However, consumer hardware typically tracks only three points: the head (HMD) and two hands (controllers). Inferring the positions of the remaining 19+ joints is an under-determined problem. Traditional approaches often struggle with the "lower body problem," where leg movements appear disconnected or unrealistic. This paper addresses these challenges by adopting a graph-based perspective, leveraging the inherent structural constraints of the human body to predict missing joint data from sparse inputs.

Methodology

The proposed framework represents the human skeleton as a Body Pose Graph (BPG). The reconstruction process is divided into three primary stages:

1. Feature Integration

Sensors provide both positional and rotational data. The model employs "interactive learning" to fuse these distinct physical properties into a unified feature set for each node, ensuring that the input to the graph is rich and multi-modal.

2. Node Property Generation

To capture the nuances of human movement, the system uses two key strategies:

  • Temporal Pyramid Structure: This component analyzes motion across multiple time scales (individual frames and short clips) to maintain motion continuity.
  • Spatial Property Modeling: Recognizing the hierarchical nature of anatomy, the model uses a unidirectional learning approach where trunk features guide the extraction of limb features.

3. Node Feature Updating with Expressive Edges

The core innovation lies in the use of Expressive Edges within a Graph Convolutional Network (GCN). Unlike standard GCNs with fixed connections, this model utilizes:

  • Static Skeleton Edges: Representing fixed physical bone connections.
  • Dynamic Skeleton Edges: Adjusting weights based on the current pose.
  • Dynamic Latent Edges: Modeling non-physical but motion-related coordination (e.g., hand-foot synchronization during locomotion).
Overview of the proposed framework, illustrating feature integration, node property generation, and the expressive edge updating mechanism.

Experimental Results

The model was trained and evaluated on the AMASS dataset (including CMU, BMLrub, and HDM05 subsets).

Quantitative Performance

The proposed method significantly outperforms previous state-of-the-art models like AvatarPoser. It achieved a Mean Per Joint Position Error (MPJPE) of 3.34 cm, a marked improvement over AvatarPoser’s 4.11 cm. The most significant gains were observed in the lower body, with an 18% to 22% improvement in accuracy for knee and foot joints.

Qualitative Realism

Visual comparisons demonstrate the model's superior ability to reconstruct complex movements. While baseline models often fail to capture the full range of motion in the legs, the BPG-based approach accurately reconstructs fluid, high-amplitude actions such as kicking.

Qualitative comparison of motion reconstruction for a kicking action, showing the improved realism and accuracy of the proposed graph-based approach.

Conclusion

By framing sparse pose estimation as a graph completion problem, this research demonstrates that structural and temporal priors can effectively overcome data sparsity. Future work will focus on improving robustness to sensor noise and optimizing the computational efficiency of dynamic GCNs for mobile VR hardware.

Summary updated

BibTeX
@inproceedings{yao2024full,
  author = {Feiyu Yao and Zongkai Wu and Li Yi},
  title = {{Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28483},
}
Figure