Towards Robust 3D Pose Transfer with Adversarial Learning

Haoyu Chen, Hao Tang, Ehsan Adeli, Guoying Zhao

Introduction

3D pose transfer—the process of applying the skeletal posture of a source subject to the body shape of a target subject—is a fundamental task in computer graphics, animation, and augmented reality. Despite its utility, the field has long been constrained by a "cleanliness" bottleneck. Most existing state-of-the-art methods rely on perfectly registered meshes, parametric models like SMPL, or clean skeletal joints.

In real-world scenarios, however, data is rarely pristine. Raw 3D scans from depth sensors are often noisy, incomplete, and lack consistent topology. Traditionally, bridging the gap between raw scans and usable 3D models requires labor-intensive manual pre-processing, which precludes real-time application. This paper introduces 3D-PoseMAE, a framework designed to perform robust pose transfer directly from raw, noisy data by leveraging multi-scale masked autoencoders and a novel adversarial training pipeline.

The 3D-PoseMAE Framework

The authors propose a two-pronged innovation to address the limitations of current 3D pose transfer models: a specialized architecture and a robustness-focused training strategy.

1. Multi-scale Masked Autoencoding

Standard 3D Masked Autoencoders (MAEs) often focus on reconstructing local geometric details. However, for pose transfer, local details like skin wrinkles or surface textures are often "noise" that can distract from the underlying skeletal structure. 3D-PoseMAE utilizes a multi-scale downsampling and masking strategy that forces the encoder to prioritize "extrinsic" attributes (the pose) over "intrinsic" local geometry.

To handle high-resolution meshes efficiently, the model replaces traditional spatial attention—which scales quadratically with the number of points ($N \times N$)—with Channel-wise Cross-Attention. By focusing on the compact latent representations ($C \times C$), the model achieves significant computational efficiency without sacrificing the global context necessary to understand complex human poses.

2. Adversarial Training for Generative Robustness

Perhaps the most significant contribution of the paper is the introduction of an adversarial training pipeline for 3D generation. While adversarial attacks are common in classification tasks, applying them to continuous generative processes like pose transfer is novel.

The authors introduce a Pose Transfer (PT) adversarial function that generates "hard" samples during the training phase. By intentionally perturbing the input and requiring the model to minimize the distance between the generated output and the ground truth, the model learns to become "invulnerable" to the types of noise and artifacts found in real-world scans. In this context, adversarial defense serves as a powerful mechanism for domain generalization.

Experimental Evaluation

The model was rigorously tested across three major datasets: SMPL-NPT (synthetic benchmarking), FAUST (generalization testing), and DFAUST (high-resolution 4D scans). The latter was particularly crucial for evaluating the model's performance on raw, unregistered data.

Key Results

  • State-of-the-Art Accuracy: On the SMPL-NPT "unseen" protocol, 3D-PoseMAE achieved a Point-wise Mesh Euclidean Distance (PMD) of 3.4, outperforming the previous leading method, GC-Transformer, which scored 4.0.
  • Resilience to Noise: In tests involving Gaussian noise, 3D-PoseMAE demonstrated remarkable stability, achieving a PMD of 6.13, whereas baseline models degraded significantly to a PMD of 12.66.
  • Raw Scan Performance: The model successfully processed raw DFAUST scans that caused previous methods to produce distorted or nonsensical results. This confirms the model's ability to bypass the manual pre-processing stage entirely.
  • Efficiency: The architecture remains lightweight with 40.7M parameters and an inference time of approximately 0.0048 seconds, making it a viable candidate for real-time AR/VR pipelines.

Discussion and Insights

The research yields several critical insights for the 3D vision community. First, it demonstrates that pose is a global attribute. By intentionally discarding local geometric details through masking and channel-wise attention, the model captures the skeletal essence more effectively. Second, it redefines the role of adversarial learning, showing it to be an effective tool for improving robustness against environmental noise rather than just a security measure.

Limitations

Despite its success, the framework faces challenges regarding training complexity. The adversarial training process is computationally intensive, requiring roughly 40 hours of training compared to 15 hours for a standard version. Additionally, the authors utilized a two-stage training approach to maintain stability. Future work may focus on stabilizing this into a single-stage process and improving the model's performance on extremely occluded or "broken" scans where significant portions of the body are missing.

Conclusion

3D-PoseMAE represents a significant step toward making 3D animation and pose transfer accessible for real-world, uncurated data. By combining efficient transformer architectures with adversarial robustness, the framework provides a path toward real-time, high-fidelity pose transfer that can operate directly on raw sensor output.

Summary updated

Abstract

3D pose transfer that aims to transfer the desired pose to a target mesh is one of the most challenging 3D generation tasks. Previous attempts rely on well-defined parametric human models or skeletal joints as driving pose sources. However, to obtain those clean pose sources, cumbersome but necessary pre-processing pipelines are inevitable, hindering implementations of the real-time applications. This work is driven by the intuition that the robustness of the model can be enhanced by introducing adversarial samples into the training, leading to a more invulnerable model to the noisy inputs, which even can be further extended to directly handling the real-world data like raw point clouds/scans without intermediate processing. Furthermore, we propose a novel 3D pose Masked Autoencoder (3D-PoseMAE), a customized MAE that effectively learns 3D extrinsic presentations (i.e., pose). 3D-PoseMAE facilitates learning from the aspect of extrinsic attributes by simultaneously generating adversarial samples that perturb the model and learning the arbitrary raw noisy poses via a multi-scale masking strategy. Both qualitative and quantitative studies show that the transferred meshes given by our network result in much better quality. Besides, we demonstrate the strong generalizability of our method on various poses, different domains, and even raw scans. Experimental results also show meaningful insights that the intermediate adversarial samples generated in the training can successfully attack the existing pose transfer models.
BibTeX
@inproceedings{chen2024towards,
  author = {Haoyu Chen and Hao Tang and Ehsan Adeli and Guoying Zhao},
  title = {{Towards Robust 3D Pose Transfer with Adversarial Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure