PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling

Zhe Li, Zerong Zheng, Yuxiao Liu, Boyao Zhou, Yebin Liu

Introduction

Creating high-fidelity, animatable digital avatars from multi-view RGB videos remains a cornerstone challenge in computer vision and graphics. The primary difficulty lies in the "low-to-high" frequency mapping problem: translating low-frequency skeletal pose data (simple joint angles) into high-frequency dynamic appearances, such as complex garment textures and shifting cloth wrinkles.

Existing methods typically rely on Coordinate-based Multi-Layer Perceptrons (MLPs) or global latent codes. However, these approaches suffer from two significant drawbacks. First, the "spectral bias" of MLPs makes it difficult for them to learn high-frequency details from low-frequency inputs, often resulting in blurry outputs. Second, global latent codes struggle with generalization; when a character performs a pose not seen during training, the model fails to extrapolate realistically, leading to artifacts or broken geometry.

The PoseVocab Framework

PoseVocab addresses these limitations by shifting the burden of learning appearance from the MLP's weights to a structured set of learnable embeddings. Instead of a single global code, the framework decomposes the human body into individual joints and builds a "vocabulary" of joint-specific latent embeddings.

Joint-structured Pose Embeddings

For each joint, the system samples "key rotations" from the training data to construct a vocabulary. Each key rotation is assigned its own learnable embedding. This approach ensures that the model learns local deformations—for instance, how a shoulder rotation specifically affects the surrounding shirt fabric—without introducing spurious correlations to unrelated body parts like the legs.

Efficient Feature Lines

To maintain spatial detail without exhausting GPU memory, PoseVocab represents each embedding as three "feature lines" projected along the x, y, and z axes. This 1D representation is significantly more compact than 3D volumes or 2D tri-planes. This efficiency allows the system to store thousands of embeddings—covering the full vocabulary of every joint—within the constraints of consumer-grade hardware.

Hierarchical Query Strategy

To render a specific pose, the system employs a three-step lookup process:

  1. Joint Level: The global pose is decomposed into individual joint rotations.
  2. Key-rotation Level: For each joint, the system identifies the $K$-nearest neighbor key rotations in the vocabulary and interpolates their embeddings. This relies on interpolation rather than extrapolation, leading to much higher stability for novel poses.
  3. Spatial Level: The specific 3D point's feature is sampled from the interpolated feature lines and fed into a pose-conditioned Neural Radiance Field (specifically VolSDF) to produce the final color and geometry.

Experimental Evaluation

The researchers evaluated PoseVocab across several prominent multi-view human datasets, including THuman4.0, DeepCap, and ZJU-MoCap. To test the model's ability to generalize to unseen movements, they used the AIST++ dataset to drive the learned avatars with complex dance sequences.

Quantitative Results

PoseVocab significantly outperforms state-of-the-art (SOTA) methods such as ARAH, TAVA, and Neural Actor. On the THuman4.0 "subject00" sequence, PoseVocab achieved a PSNR of 30.97 on novel poses, a substantial improvement over ARAH (21.76) and TAVA (26.60). Furthermore, the model showed a massive improvement in LPIPS (Learned Perceptual Image Patch Similarity), indicating that the visual results are much sharper and more perceptually accurate.

Qualitative Performance

Visual comparisons highlight PoseVocab's ability to maintain crisp details, such as distinct logos on t-shirts and sharp wrinkles on hoodies, which remain stable during motion. While competing methods often produce blurry textures or "ghosting" artifacts during complex movements, PoseVocab maintains structural integrity and realistic cloth deformation even when driven by the "fancy" dance poses of the AIST++ dataset.

Conclusion and Future Work

PoseVocab introduces a scalable and efficient method for encoding human pose that prioritizes local disentanglement and interpolation. By utilizing a joint-specific vocabulary and 1D feature lines, it achieves superior fidelity and generalization compared to previous MLP-based approaches.

Despite its success, the method has limitations. It currently relies on SMPL-based inverse skinning, which makes it less effective for "loose" clothing like long dresses or skirts that do not strictly follow bone movement. Additionally, the current implementation does not focus on high-fidelity finger movements or facial expressions. Future work may involve integrating PoseVocab into specialized representations for loose garments to further enhance the realism of digital humans.

Summary updated

BibTeX
@inproceedings{li2023posevocab,
  author = {Zhe Li and Zerong Zheng and Yuxiao Liu and Boyao Zhou and Yebin Liu},
  title = {{PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2023},
  doi = {10.1145/3588432.3591490},
}
Figure