Introduction
3D pose transfer—the process of applying the skeletal posture of a source subject to the body shape of a target subject—is a fundamental task in computer graphics, animation, and augmented reality. Despite its utility, the field has long been constrained by a "cleanliness" bottleneck. Most existing state-of-the-art methods rely on perfectly registered meshes, parametric models like SMPL, or clean skeletal joints.
In real-world scenarios, however, data is rarely pristine. Raw 3D scans from depth sensors are often noisy, incomplete, and lack consistent topology. Traditionally, bridging the gap between raw scans and usable 3D models requires labor-intensive manual pre-processing, which precludes real-time application. This paper introduces 3D-PoseMAE, a framework designed to perform robust pose transfer directly from raw, noisy data by leveraging multi-scale masked autoencoders and a novel adversarial training pipeline.
The 3D-PoseMAE Framework
The authors propose a two-pronged innovation to address the limitations of current 3D pose transfer models: a specialized architecture and a robustness-focused training strategy.
1. Multi-scale Masked Autoencoding
Standard 3D Masked Autoencoders (MAEs) often focus on reconstructing local geometric details. However, for pose transfer, local details like skin wrinkles or surface textures are often "noise" that can distract from the underlying skeletal structure. 3D-PoseMAE utilizes a multi-scale downsampling and masking strategy that forces the encoder to prioritize "extrinsic" attributes (the pose) over "intrinsic" local geometry.
To handle high-resolution meshes efficiently, the model replaces traditional spatial attention—which scales quadratically with the number of points ($N \times N$)—with Channel-wise Cross-Attention. By focusing on the compact latent representations ($C \times C$), the model achieves significant computational efficiency without sacrificing the global context necessary to understand complex human poses.
2. Adversarial Training for Generative Robustness
Perhaps the most significant contribution of the paper is the introduction of an adversarial training pipeline for 3D generation. While adversarial attacks are common in classification tasks, applying them to continuous generative processes like pose transfer is novel.
The authors introduce a Pose Transfer (PT) adversarial function that generates "hard" samples during the training phase. By intentionally perturbing the input and requiring the model to minimize the distance between the generated output and the ground truth, the model learns to become "invulnerable" to the types of noise and artifacts found in real-world scans. In this context, adversarial defense serves as a powerful mechanism for domain generalization.
Experimental Evaluation
The model was rigorously tested across three major datasets: SMPL-NPT (synthetic benchmarking), FAUST (generalization testing), and DFAUST (high-resolution 4D scans). The latter was particularly crucial for evaluating the model's performance on raw, unregistered data.
Key Results
- State-of-the-Art Accuracy: On the SMPL-NPT "unseen" protocol, 3D-PoseMAE achieved a Point-wise Mesh Euclidean Distance (PMD) of 3.4, outperforming the previous leading method, GC-Transformer, which scored 4.0.
- Resilience to Noise: In tests involving Gaussian noise, 3D-PoseMAE demonstrated remarkable stability, achieving a PMD of 6.13, whereas baseline models degraded significantly to a PMD of 12.66.
- Raw Scan Performance: The model successfully processed raw DFAUST scans that caused previous methods to produce distorted or nonsensical results. This confirms the model's ability to bypass the manual pre-processing stage entirely.
- Efficiency: The architecture remains lightweight with 40.7M parameters and an inference time of approximately 0.0048 seconds, making it a viable candidate for real-time AR/VR pipelines.
Discussion and Insights
The research yields several critical insights for the 3D vision community. First, it demonstrates that pose is a global attribute. By intentionally discarding local geometric details through masking and channel-wise attention, the model captures the skeletal essence more effectively. Second, it redefines the role of adversarial learning, showing it to be an effective tool for improving robustness against environmental noise rather than just a security measure.
Limitations
Despite its success, the framework faces challenges regarding training complexity. The adversarial training process is computationally intensive, requiring roughly 40 hours of training compared to 15 hours for a standard version. Additionally, the authors utilized a two-stage training approach to maintain stability. Future work may focus on stabilizing this into a single-stage process and improving the model's performance on extremely occluded or "broken" scans where significant portions of the body are missing.
Conclusion
3D-PoseMAE represents a significant step toward making 3D animation and pose transfer accessible for real-world, uncurated data. By combining efficient transformer architectures with adversarial robustness, the framework provides a path toward real-time, high-fidelity pose transfer that can operate directly on raw sensor output.
Summary updated