TL;DR
This paper introduces a robust framework for 3D pose transfer that can handle messy, real-world data like raw scans and noisy point clouds without needing "clean" pre-processed inputs. By combining a novel 3D-Pose Masked Autoencoder (3D-PoseMAE) with an on-the-fly adversarial training pipeline, the authors achieve state-of-the-art performance and bridge the gap between synthetic training data and challenging real-world captures.
What Problem Does This Paper Address?
3D pose transfer is the task of taking the "pose" from one source (like a person waving) and applying it to a different "identity" mesh (like a 3D model of a different person) while keeping that person's unique body shape.
Most existing methods are "fair-weather" models: they work beautifully on clean, synthetic meshes but fall apart when faced with real-world data. Real-world 3D data is often noisy, incomplete, or comes as raw point clouds from sensors. To use current models, researchers usually have to go through a "cumbersome but necessary" pre-processing pipeline—like registering raw scans to a parametric model (e.g., SMPL)—which can take minutes per frame. This makes real-time applications in AR/VR or gaming nearly impossible. The authors aim to build a model that is robust enough to skip these steps and work directly on raw, noisy inputs.
Methods
The authors tackle robustness from two angles: architecture and training strategy.
- 3D-PoseMAE Architecture: They propose a customized Masked Autoencoder (MAE) designed specifically for pose.
- Multi-scale Masking: Instead of just masking random points, they downsample the source pose at multiple scales and mask them. This forces the model to learn a consistent, global "extrinsic attribute" (the pose) that is shared across all scales, making it less sensitive to local noise.
- Channel-wise Cross-Attention: Traditional 3D transformers use spatial attention, which can be computationally heavy and focus too much on redundant geometric details (like wrinkles). This model uses channel-wise attention to integrate pose and identity information in a more compact latent space, improving both speed and quality.
- Adversarial Training Pipeline: To make the model "invulnerable," they don't just train on clean data. They implement an on-the-fly adversarial attack during training.
- They define a new Pose Transfer (PT) adversarial function that calculates how to perturb the input pose to maximize the error in the output.
- By training against these "hard" adversarial samples, the model learns to ignore perturbations and focus on the underlying pose structure.
Datasets
The model was trained and evaluated using several key datasets:
- SMPL-NPT: A large synthetic dataset of 24,000 body meshes used for primary training and quantitative benchmarking.
- FAUST: A dataset of real human body scans used to test how well the model generalizes to real-world identities.
- DFAUST: High-resolution 4D scans used specifically to evaluate the model's ability to handle raw, incomplete scans from the real world.
Key Insights and Ideas
- Adversarial Training as a Bridge: The most striking insight is that adversarial training isn't just for security; it's a powerful tool for domain adaptation. By training on "attacked" synthetic data, the model naturally becomes better at handling the "natural attacks" found in real-world raw scans (noise, holes, outliers).
- Pose vs. Geometry: The authors argue that for pose transfer, local geometric details are often "redundant information." By using channel-wise attention and multi-scale masking, they effectively filter out this noise, allowing the model to focus on the skeletal-level configuration of the body.
- On-the-fly Attacks: Unlike previous 3D adversarial methods that used pre-computed samples, this framework generates attacks during the backpropagation step of training. This ensures the model is always being challenged by the most effective perturbations for its current state.
Key Results
- State-of-the-Art Accuracy: On clean data (SMPL-NPT), 3D-PoseMAE achieved a Point-wise Mesh Euclidean Distance (PMD) of 0.6 on seen poses and 3.4 on unseen poses, outperforming previous leaders like GC-Transformer.
- Superior Robustness: When tested against adversarial attacks, 3D-PoseMAE maintained a PMD of 16.9, while a standard GC-Transformer's error skyrocketed to 105.2.
- Real-World Performance: On raw scans from DFAUST, the model achieved a PMD of 13.70, nearly half the error of the next best baseline (25.21).
- Efficiency: Despite its robustness, the model is relatively lightweight (~40.7M parameters) and fast, with an inference time of roughly 4.8ms per frame.
Limitations and Future Work
The authors note that the balance between attack magnitude and model robustness is delicate. If adversarial samples are introduced too early in training, the model can struggle to converge and fall into local minima. Consequently, they utilize a two-stage training process (200 epochs clean, 200 epochs adversarial).
Future work could explore extending this robustness to even more extreme data types, such as heavily occluded scenes or non-human characters, and further optimizing the on-the-fly attack generation to reduce the overall training time (which currently takes about 40 hours).
Summary updated