Generative models have become remarkably adept at "imagining" human appearances, but they often struggle with the underlying physics of the human body. When left to their own devices, unconstrained models—whether based on Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), or Diffusion—frequently produce "melting" limbs, floating artifacts, or the notorious "Janus problem," where a 3D model might have multiple faces.
The solution emerging across recent research is to use parametric 3D models (such as SMPL for bodies, MANO for hands, or 3DMM for faces) as a structural scaffold. By anchoring high-dimensional generative processes to these low-dimensional, anatomically-correct templates, researchers can ensure physical plausibility while maintaining the expressive freedom of modern generative AI.
The Problem of High-Dimensional Freedom
Generating a 3D human from a single image or text prompt is an under-determined problem. A model must hallucinate the back of a person, the exact fold of a sleeve, and the precise articulation of a finger. Purely 2D-to-3D methods often fail because they lack a "sense" of anatomy. As Shanyan Guan and colleagues note in Human Action Transfer Based on 3D Model Reconstruction, 2D generative models struggle with consistent proportions, leading to distortions during motion.
By introducing a parametric model, the search space is reduced from millions of independent pixels or voxels to a compact set of shape and pose parameters. This structural prior acts as a "geometric anchor," preventing the model from producing physically impossible configurations.
Structural Initialization: Building on a Skeleton
One of the most effective ways to use these priors is during the initialization phase. Instead of starting from random noise or a sparse point cloud, the representation is "seeded" using the geometry of a parametric mesh.
In the realm of 3D Gaussian Splatting, Xian Liu and Ying Shan demonstrate in HumanGaussian that sampling points directly from an SMPL-X mesh provides a reliable anatomical foundation. This ensures the model starts with correct proportions, allowing the optimization to focus entirely on fine-grained details like clothing wrinkles and accessories. Similarly, Hyunjun Jung and the team behind ParDy-Human initialize Gaussians on the vertices of a canonical mesh, ensuring the representation is pre-aligned with a known skeletal structure.
Constrained Refinement and Hallucination
Once initialized, the parametric model serves as a "soft" constraint. It provides the coarse skeleton, but the generative model is allowed to deviate from it to capture non-body details like hair or loose clothing.
Yangyi Huang and Hongwei Yi use this approach in ELICIT, where a 3D human template prevents the representation from collapsing into a flat structure during one-shot reconstruction. This "scaffolding" is particularly vital for hallucinating occluded parts. For instance, Junying Wang shows that integrating a parametric body model allows a system to reconstruct a full 3D human even when the input image is incomplete, as the prior provides the necessary "logic" for where the hidden limbs should be.
Cross-Modal Guidance for Diffusion
For diffusion-based models, the parametric prior often takes the form of a conditioning signal. Rather than asking a model to "draw a hand," researchers like Zhenyue Qin in HandCraft use a 3D parametric hand template to generate precise depth and mask signals. These signals act as a blueprint, forcing the diffusion model to adhere to correct anatomical proportions while it fills in the textures.
This "blueprint" approach is also central to PSHuman by Peng Li and Yan-Pei Cao. By rendering the parametric model and feeding it as a condition, the generative model is anchored to a physically plausible pose, preventing the "hallucination" of unnatural limb configurations during multiview synthesis.
Why It Works: The Efficiency of Priors
The recurring theme across these 48 papers is that learning from scratch is inefficient. By baking millions of years of human evolution (in the form of anatomical constraints) into the neural architecture, models can achieve higher fidelity with less data and faster training times.
As Shoukang Hu notes in GauHuman, using structural templates allows optimization to bypass the discovery of basic body topology. This shifts the computational burden from "figuring out what a human looks like" to "learning what this specific human looks like," leading to the real-time, high-fidelity digital twins that are now becoming possible.