Towards Robust 3D Pose Transfer with Adversarial Learning

Haoyu Chen, Hao Tang, Ehsan Adeli, Guoying Zhao

TL;DR

This paper introduces a robust framework for 3D pose transfer that can handle messy, real-world data like raw scans and noisy point clouds without needing "clean" pre-processed inputs. By combining a novel 3D-Pose Masked Autoencoder (3D-PoseMAE) with an on-the-fly adversarial training pipeline, the authors achieve state-of-the-art performance and bridge the gap between synthetic training data and challenging real-world captures.

What Problem Does This Paper Address?

3D pose transfer is the task of taking the "pose" from one source (like a person waving) and applying it to a different "identity" mesh (like a 3D model of a different person) while keeping that person's unique body shape.

Most existing methods are "fair-weather" models: they work beautifully on clean, synthetic meshes but fall apart when faced with real-world data. Real-world 3D data is often noisy, incomplete, or comes as raw point clouds from sensors. To use current models, researchers usually have to go through a "cumbersome but necessary" pre-processing pipeline—like registering raw scans to a parametric model (e.g., SMPL)—which can take minutes per frame. This makes real-time applications in AR/VR or gaming nearly impossible. The authors aim to build a model that is robust enough to skip these steps and work directly on raw, noisy inputs.

Figure 1. Examples of our 3D pose transfer results on various pose sources, show strong robustness and generalizability. The pose source includes clean mesh (top left) and point clouds with Gaussian noise (bottom left) from SMPL-NPT dataset [41], the adversarial sample of point cloud generated by our method (top right), and raw scan (bottom right) from DFAUST dataset [5]. Identity meshes are from the SMPL-NPT dataset [41] and the FAUST [3] (bottom right) dataset. Our method can achieve promising pose transfer performance even on the extremely challenging incomplete raw scan (bottom right). See more results and details in the Supplementary Materials.

Methods

The authors tackle robustness from two angles: architecture and training strategy.

  1. 3D-PoseMAE Architecture: They propose a customized Masked Autoencoder (MAE) designed specifically for pose.
  2. Multi-scale Masking: Instead of just masking random points, they downsample the source pose at multiple scales and mask them. This forces the model to learn a consistent, global "extrinsic attribute" (the pose) that is shared across all scales, making it less sensitive to local noise.
  3. Channel-wise Cross-Attention: Traditional 3D transformers use spatial attention, which can be computationally heavy and focus too much on redundant geometric details (like wrinkles). This model uses channel-wise attention to integrate pose and identity information in a more compact latent space, improving both speed and quality.
  4. Adversarial Training Pipeline: To make the model "invulnerable," they don't just train on clean data. They implement an on-the-fly adversarial attack during training.
  5. They define a new Pose Transfer (PT) adversarial function that calculates how to perturb the input pose to maximize the error in the output.
  6. By training against these "hard" adversarial samples, the model learns to ignore perturbations and focus on the underlying pose structure.
Figure 2. Left: The existing methods can deal with simple perturbations such as Gaussian noises but fail to handle harder inputs in real- world cases. Middle: The traditional pipeline used in previous methods [10, 27, 37, 41] for 3D pose transfer. The model is trained with clean mesh inputs without considering the robustness to the noisy inputs. We use the symbol ∼to generally refer to the loss term, which differs according to the actual condition. Right: Our method. Our method utilizes the strength of adversarial learning to enhance the robustness and generalizability of the model. It consists of an adversarial sample generating flow(top part in red) and a pose transferring flow(bottom part in green). The two flows happen iteratively during the adversarial training and the adversarial samples are calculated on-the-fly. Note that Mid, Mpose, Mresult, and MGT stand for the identity, pose, generated meshes, and ground truths, the same as below.

Datasets

The model was trained and evaluated using several key datasets:

  • SMPL-NPT: A large synthetic dataset of 24,000 body meshes used for primary training and quantitative benchmarking.
  • FAUST: A dataset of real human body scans used to test how well the model generalizes to real-world identities.
  • DFAUST: High-resolution 4D scans used specifically to evaluate the model's ability to handle raw, incomplete scans from the real world.

Key Insights and Ideas

  • Adversarial Training as a Bridge: The most striking insight is that adversarial training isn't just for security; it's a powerful tool for domain adaptation. By training on "attacked" synthetic data, the model naturally becomes better at handling the "natural attacks" found in real-world raw scans (noise, holes, outliers).
  • Pose vs. Geometry: The authors argue that for pose transfer, local geometric details are often "redundant information." By using channel-wise attention and multi-scale masking, they effectively filter out this noise, allowing the model to focus on the skeletal-level configuration of the body.
  • On-the-fly Attacks: Unlike previous 3D adversarial methods that used pre-computed samples, this framework generates attacks during the backpropagation step of training. This ensures the model is always being challenged by the most effective perturbations for its current state.

Key Results

  • State-of-the-Art Accuracy: On clean data (SMPL-NPT), 3D-PoseMAE achieved a Point-wise Mesh Euclidean Distance (PMD) of 0.6 on seen poses and 3.4 on unseen poses, outperforming previous leaders like GC-Transformer.
  • Superior Robustness: When tested against adversarial attacks, 3D-PoseMAE maintained a PMD of 16.9, while a standard GC-Transformer's error skyrocketed to 105.2.
  • Real-World Performance: On raw scans from DFAUST, the model achieved a PMD of 13.70, nearly half the error of the next best baseline (25.21).
  • Efficiency: Despite its robustness, the model is relatively lightweight (~40.7M parameters) and fast, with an inference time of roughly 4.8ms per frame.

Limitations and Future Work

The authors note that the balance between attack magnitude and model robustness is delicate. If adversarial samples are introduced too early in training, the model can struggle to converge and fall into local minima. Consequently, they utilize a two-stage training process (200 epochs clean, 200 epochs adversarial).

Future work could explore extending this robustness to even more extreme data types, such as heavily occluded scenes or non-human characters, and further optimizing the on-the-fly attack generation to reduce the overall training time (which currently takes about 40 hours).

Summary updated

Abstract

3D pose transfer that aims to transfer the desired pose to a target mesh is one of the most challenging 3D generation tasks. Previous attempts rely on well-defined parametric human models or skeletal joints as driving pose sources. However, to obtain those clean pose sources, cumbersome but necessary pre-processing pipelines are inevitable, hindering implementations of the real-time applications. This work is driven by the intuition that the robustness of the model can be enhanced by introducing adversarial samples into the training, leading to a more invulnerable model to the noisy inputs, which even can be further extended to directly handling the real-world data like raw point clouds/scans without intermediate processing. Furthermore, we propose a novel 3D pose Masked Autoencoder (3D-PoseMAE), a customized MAE that effectively learns 3D extrinsic presentations (i.e., pose). 3D-PoseMAE facilitates learning from the aspect of extrinsic attributes by simultaneously generating adversarial samples that perturb the model and learning the arbitrary raw noisy poses via a multi-scale masking strategy. Both qualitative and quantitative studies show that the transferred meshes given by our network result in much better quality. Besides, we demonstrate the strong generalizability of our method on various poses, different domains, and even raw scans. Experimental results also show meaningful insights that the intermediate adversarial samples generated in the training can successfully attack the existing pose transfer models.
BibTeX
@inproceedings{chen2024towards,
  author = {Haoyu Chen and Hao Tang and Ehsan Adeli and Guoying Zhao},
  title = {{Towards Robust 3D Pose Transfer with Adversarial Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure