TL;DR
PoseMaster is a unified, 3D-native generative framework that achieves precise 3D character pose stylization directly from a single reference image and a target 3D skeleton. By replacing traditional multi-stage 2D-to-3D cascade pipelines with a single-stage flow-based Diffusion Transformer, it eliminates error propagation and depth ambiguity. Crucially, because the generated 3D meshes strictly align with the input 3D skeleton, PoseMaster enables the immediate creation of fully rigged, animatable assets.
What Problem Does This Paper Address?
Generating stylized 3D characters that match a specific target pose while preserving the visual identity of a reference image is a major bottleneck in gaming, film, and VR.
Existing state-of-the-art methods typically rely on a decoupled cascade pipeline:
- Use a 2D controllable model (like ControlNet) guided by a 2D skeleton image to manipulate the character's pose in 2D space.
- Use a separate 3D reconstruction model to "lift" that generated 2D image into a 3D asset.
This approach suffers from two fundamental flaws. First, error propagation is inevitable—any artifacts, warping, or self-occlusion mistakes made by the 2D model are permanently baked into and amplified by the 3D reconstruction. Second, 2D skeleton projections introduce geometric ambiguity, as flattening a pose into 2D strips away critical depth and topological information.
PoseMaster bypasses these limitations by introducing a 3D-native framework that handles pose control and 3D generation simultaneously.
Methods
PoseMaster unifies pose stylization and 3D generation into an end-to-end architecture consisting of a 3D Variational Autoencoder (VAE) and a flow-based Diffusion Transformer (DiT), built upon the Hunyuan3D 2.1 backbone.
The core technical innovations include:
- 3D Skeleton Control Mechanism: Instead of 2D maps, the model uses explicit 3D spatial coordinates.
- Skeleton Densification Strategy: Standard 3D skeletons only define sparse joints, which lacks topological connectivity. PoseMaster uses a distance-weighted interpolation scheme to transform bones into a dense point cloud. It then embeds explicit 3D bone directional vectors into these points to give the network structural awareness.
- Point Transformer Skeleton Encoder: This module processes the densified coordinates and directional features into a high-dimensional skeleton condition token ($c_p$).
- Multi-Condition DiT: The model injects both the image condition ($c_i$, extracted via DINOv2) and the skeleton condition ($c_p$) at the token level into the DiT block, using Classifier-Free Guidance (CFG) to balance identity preservation and pose adherence.
Datasets
Training a 3D-native model requires massive amounts of paired data, which is notoriously scarce. To solve this, the authors built a scalable Data Engine to curate a large-scale dataset of "Image-Skeleton-Mesh" triplets:
- Animatable 3D Assets: Sourced from ReadyPlayerMe, VRoid, and Playbox. Characters were animated using motion sequences to create Action Pairs—sampling an image from one frame and pairing it with the mesh and skeleton from a completely different frame to force pose decoupling.
- Static 3D Meshes: Sourced from Objaverse, Objaverse-XL, and HumanRig. For unrigged models, an automatic rigging model was used to extract skeletons. Multi-view renderings were generated to create View Pairs, forcing the network to infer occluded regions.
In total, the data engine produced data for over 500K unique humanoid objects, scaling into millions of total training pairs.
Key Insights and Ideas
- The 3D Skeleton as a Geometric Anchor: Directly feeding a 3D skeleton into a 3D native generator acts as an explicit structural scaffold. This resolves the monocular depth ambiguity that plagues standard image-to-3D models, preventing geometric collapse in complex or self-occluded poses.
- Bypassing the 2D Intermediate Step: By eliminating the intermediate 2D image-editing step, the model avoids the structural distortions common to 2D models, resulting in significantly cleaner mesh topologies.
- Downstream Rigging for Free: Because the generated 3D mesh is synthesized to strictly conform to the input 3D skeleton's spatial layout, the asset is inherently pre-aligned. When paired with standard automated skinning models, it can be animated immediately, completely skipping manual skeleton retargeting.
Key Results
The authors evaluated PoseMaster on two primary tasks: pose canonicalization (turning an arbitrary pose into a standard T/A-pose) and arbitrary-pose stylization.
Quantitative Performance
Evaluated using Mean Angular Error (MAE) and Cosine Similarity (Sim) of surface normals to circumvent spatial misalignment penalties:
- Pose Canonicalization (VRoid dataset): PoseMaster achieved an MAE of 4.59 and Sim of 0.938, outperforming task-specific pipelines like StdGen (MAE 4.97) and native generators like Hunyuan3D 2.1 (MAE 5.89).
- Arbitrary-Pose Stylization: PoseMaster achieved an MAE of 5.28, significantly lower than Hunyuan3D 2.1 (6.75) and Trellis (7.20), even when those baselines were given the ground-truth target image as input.
Efficiency
Because it is a single-stage framework, PoseMaster cuts down intermediate computational bottlenecks, achieving an inference time of ~23.48 seconds on a single NVIDIA H20 GPU, compared to StdGen's ~61.54 seconds.
Limitations and Future Work
While PoseMaster demonstrates strong overall pose controllability, the authors note a few limitations:
- Fine-Grained Details: The model still struggles to synthesize highly intricate geometric details, such as complex hairstyles, flowing skirts, or precise finger/hand gestures.
- Resolution Limits: The current single-stage generation paradigm operates at a spatial resolution of 512. As a result, its high-frequency geometric fidelity can lag behind multi-stage refinement methods.
The authors suggest that integrating a high-resolution geometric refinement module in future work will be key to upscaling the coarse meshes and achieving photorealistic details.
Summary updated