When animating a digital human, hand, or garment, traditional computer graphics relies on a skeletal rig to drive a surface mesh. However, standard techniques like Linear Blend Skinning (LBS) assume a static relationship between bones and surface vertices. As noted by Golyanik et al., this linear assumption fails to capture highly non-linear physical phenomena such as muscle bulging, skin stretching, or fabric folding.
When modern neural rendering frameworks (such as Neural Radiance Fields or 3D Gaussian Splatting) attempt to reconstruct moving subjects from video, ignoring these non-linear deformations leads to a fundamental mathematical issue: a "one-to-many" mapping problem. A single spatial coordinate in a character's local space can experience completely different wrinkle patterns or geometric configurations across different frames. As Liu et al. explain, without pose context, optimization forces the network to regress to a blurry, ghosted "mean appearance," failing to represent sharp, high-frequency details like folding fabric.
To resolve this ambiguity, the field of pose-conditioned appearance modeling injects the low-dimensional kinematic state of the subject directly into the neural networks predicting geometry, skinning, and shading.
Resolving Ambiguity via Pose Conditioning
The core intuition of this approach is to transform a static neural representation $f(x) \to \text{appearance}$ into a dynamic, state-aware function $f(x, \theta)$, where $\theta$ represents the skeletal joint angles or pose parameters.
By explicitly conditioning the network on the pose, the model can resolve the one-to-many mapping problem. As demonstrated by Xu et al. in H-NeRF, conditioning neural radiance and geometry networks on skeletal pose parameters allows the system to map specific joint configurations directly to localized clothing folds and surface variations.
This conditioning operates across various structural levels:
- Implicit Shapes: Dong et al. feed pose information as a conditional input to an implicit shape network, allowing the model to learn individual, non-rigid clothing behaviors without relying on generic templates.
- Explicit Offsets: For real-time hand avatars, Gan et al. utilize pose-dependent feature embeddings to predict vertex displacements and skinning weight offsets, capturing fine-grained skin stretching and compression.
Disentangling Canonical Identity from Dynamic Offsets
Feeding a raw, high-dimensional pose vector directly into a global network can make optimization difficult and harm generalization to unseen poses. To make this mapping tractable, modern architectures decouple the problem into two parts: a canonical space that captures the subject's intrinsic, unposed identity, and a pose-dependent deformation field that applies localized corrections.
Skeletal Pose (θ) ───► [Deformation Network] ───► Geometric/Color Offsets (Δx, Δc)
│
▼
Spatial Coordinate (x) ──────────────────────────► [Canonical Field] ───► Final Render
The authors of GaussianAvatar implement this by using a dynamic appearance network to predict geometry and color offsets for 3D Gaussian primitives based on the driving pose. This allows the underlying primitives to shift, scale, and recolor dynamically, preserving structural integrity during complex motions.
Furthermore, fabric and loose clothing often exhibit secondary motion that depends on movement history rather than just the instantaneous joint configuration. To capture these temporal dependencies, Choi et al. condition their skinning weight encoder on a temporal sequence of poses. Similarly, Renke Wang demonstrates that clothing structure and dynamic wrinkles can be inferred directly from a body's velocity and surface geometry history, bypassing the need for explicit, computationally expensive physics simulations.
Feature Vocabularies and Interpolation Spaces
As the complexity of the performed motions increases, mapping a continuous pose vector to high-frequency details can suffer from over-smoothing. To counteract this, researchers utilize structured latent spaces that discretize or anchor pose-dependent details.
One approach is to maintain a dictionary of localized, pose-aware features. Xiao et al. utilize a dictionary of pose-aware feature grids, allowing the model to interpolate between known geometric states to synthesize realistic movement-dependent details. Expanding on this concept, Liu et al. introduce a "texture vocabulary" that retrieves and interpolates texture features from a library of previously observed pose-appearance pairs, providing a direct lookup mechanism to handle highly non-linear appearance changes.
Alternatively, the deformation field can be parameterized using a low-dimensional basis. Kimura et al. map skeletal joint rotation matrices to the coefficients of a learned "eigen-deformation" basis. This mathematical constraint allows the network to robustly interpolate and extrapolate surface details, enabling it to hallucinate realistic clothing folds even in occluded or unobserved body regions.
Pose-Aware Shading and Relighting
A change in pose does not merely alter geometry; it fundamentally changes how a surface interacts with light. As joints bend and fabric folds, surfaces experience dynamic self-shadowing, shifting highlights, and changing occlusions.
To separate intrinsic material properties from these movement-dependent lighting changes, Li et al. introduce a pose-dependent shading model. By conditioning the radiance field on the joint configuration, the canonical representation remains dedicated to storing intrinsic albedo, while a secondary shading model handles dynamic shadows and highlights during motion.
For physically-based relighting, surface normals must be updated accurately to reflect both global skeletal rotations and local non-rigid deformations. As shown by the authors of Neural Reconstruction of Relightable Human Model from Monocular Video, explicitly transforming surface normals by both skeletal and non-rigid components ensures that light interacts realistically with the surface as the body deforms. This principle applies beyond organic forms; Liu et al. employ pose-conditioned appearance deformation to account for self-shadowing, occlusions, and mechanical deformations in articulated robotic systems, demonstrating that conditioning appearance on kinematic state is a universal requirement for high-fidelity dynamic modeling.
Key Foundational Work
Controlling deformable material with dynamic morph targets (2009) — contributed 3 concept elements to this direction.