Dynamic Mesh Recovery from Partial Point Cloud Sequence

Hojun Jang, Minkwan Kim, Jinseok Bae, Young Min KimICCV 2023 · DOI

Abstract

Recovering 3D dynamic meshes from real-world sensor data is challenging due to noise, occlusion, and temporal gaps. This paper presents a robust framework that combines a Transformer-based architecture with a Conditional Variational Autoencoder (CVAE) to reconstruct full human or hand meshes from incomplete point cloud sequences. By pre-training a "Kinematics Learner" on clean data to establish a motion prior and subsequently training a "Feature Follower" to map noisy observations into this latent space, the model achieves smooth, anatomically plausible reconstructions. Experimental results demonstrate superior accuracy and sub-second processing speeds, significantly outperforming existing optimization-based and frame-wise methods.

Introduction

Capturing 3D human motion is a cornerstone for applications in virtual reality, human-robot interaction, and motion analysis. While commodity depth sensors like LiDAR provide accessible 3D point cloud data, these observations are frequently degraded by noise, self-occlusion, and intermittent sensor failures. Existing recovery methods often struggle with temporal jitter or prohibitive computational costs. This work addresses these issues by proposing a fast, sequence-based recovery system that leverages learned kinematic priors to "fill in" missing information from messy real-world data.

Methodology

The proposed framework utilizes a two-stage learning strategy to decouple the understanding of human motion from the noise characteristics of specific sensors.

Stage 1: The Kinematics Learner

The model first learns a robust representation of human motion from large-scale datasets of complete 3D sequences (e.g., AMASS). A Feature Encoder uses PointNet for per-frame spatial features and a Transformer to capture temporal dependencies. Central to this stage is a Conditional Variational Autoencoder (CVAE), which learns a latent distribution of possible poses. This probabilistic approach allows the model to represent the inherent uncertainty in motion recovery.

Stage 2: The Feature Follower

To handle real-world data, the authors introduce a Feature Follower. This module is trained to map features from noisy, partial point clouds into the high-quality latent space defined by the Kinematics Learner. This alignment ensures that even when the input is severely degraded, the recovered mesh follows a physically plausible motion trajectory.

The framework architecture featuring the Kinematics Learner and Feature Follower.

Experiments and Results

The model was evaluated on diverse datasets, including AMASS and SURREAL for training, and Berkeley MHAD for real-world testing. It also generalizes to hand motion using the HanCo and InterHand2.6M datasets.

Performance and Speed

On the Berkeley MHAD benchmark, the framework achieved a Mean Chamfer Distance of 0.0027, a significant improvement over previous baselines like VoteHMR (0.0051). Crucially, the model operates in sub-second time, whereas traditional optimization-based approaches can take several minutes per sequence.

Robustness to Ambiguity

A key advantage of the CVAE-based architecture is its ability to generate multiple plausible reconstructions from ambiguous input. For instance, if a sensor only captures the upper body, the model can sample various anatomically correct lower-body configurations.

Qualitative results showing multiple plausible mesh samples reconstructed from a partial point cloud input.

Discussion and Key Insights

The success of this approach hinges on two main ideas:

  1. Latent Space Alignment: By fixing the Kinematics Learner and only adapting the Feature Follower, the model maintains a deep understanding of human physics while becoming robust to sensor-specific noise.
  2. Temporal Context: The Transformer's attention mechanism allows the model to use information from adjacent frames to infer the position of occluded body parts, effectively "hallucinating" missing data based on motion trends.

Limitations

Despite its strengths, the model can be sensitive to environmental clutter (e.g., floors or furniture) if not properly filtered. Future work may involve integrating RGB data to resolve further ambiguities and developing subject-specific models to capture individual movement nuances.

Conclusion

This paper introduces a highly efficient and robust solution for dynamic mesh recovery. By leveraging temporal Transformers and probabilistic latent spaces, the framework bridges the gap between messy raw sensor data and clean, usable 3D motion sequences, paving the way for real-time applications in unconstrained environments.

Summary updated

BibTeX
@inproceedings{jang2023dynamic,
  author = {Hojun Jang and Minkwan Kim and Jinseok Bae and Young Min Kim},
  title = {{Dynamic Mesh Recovery from Partial Point Cloud Sequence}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01384},
}
Figure