Animatable avatars are 3D models of a person, learned from video, that can be rendered in new body poses. Their main selling point is the case we cannot measure: putting the person in a pose they never performed on camera. Current benchmarks score novel-pose quality by comparing renders against held-out frames from the same capture session. Those frames come from the same motion repertoire as the training frames, so they are close to the training distribution. They are also scored with pixel or ImageNet-feature metrics, which punish a plausible fold of fabric for being in the wrong place and do not notice a melted elbow. For poses that are truly out of distribution, such as a dance move from another dataset applied to a ZJU-MoCap subject, no ground-truth image exists at all. Papers either show qualitative figures or compute FID-style distribution scores. So the regime that motivates the field is the one it cannot evaluate.
We propose a reference-free score for novel-pose renders. It splits quality into two parts that can each be checked without a paired target. Structure is checked against the driving pose itself. Appearance is checked against the subject's own training footage after warping back to a pose-independent body frame. We then validate the score the way self-supervised inverse-problem work validates losses that have no ground truth: on a held-out split of the training data where ground truth does exist, the score must agree with true error, and on genuinely out-of-distribution poses it must agree with human judgments better than the metrics in current use.
Core Claim
Self-supervised MRI and imaging work shows that when clean targets are missing, the observations can be split into a fitting subset and a checking subset. With the right structure, the checking subset gives a trustworthy, even unbiased, estimate of supervised error. Zero-Shot Self-Supervised Learning for MRI Reconstruction uses a held-out measurement subset for automatic early stopping. Equivariant Splitting proves that splitting combined with a known symmetry gives an unbiased estimate of the supervised loss.
Our claim is that avatars come with an analogous structure, of an articulated rather than global kind. Under linear blend skinning (the standard rule that moves each surface point with a weighted mix of nearby bones), a person's base appearance in the pose-independent canonical frame should not change with pose. Only a pose-dependent residual should change: wrinkles, shading, and secondary motion. The posed skeleton also fully specifies where limbs and the silhouette must be.
This lets novel-pose quality be scored in two ways. Rigid structure is compared to the known pose. Canonical appearance is compared to the distribution of colors that the same surface point showed across all training frames, with tolerance set by how much that point actually varied. Wrinkles at new but plausible locations stay inside the observed variation and are not penalized. Holes, ghosting, color bleeding, and limb tearing fall outside it and are penalized.
The new elements are a pose-aware, reference-free score for avatars, and a splitting-based protocol for showing that the score is trustworthy beyond the training distribution.
Research Program
The score has a structure term and an appearance term.
The structure term uses detectors that played no part in training the avatar: an off-the-shelf 2D keypoint detector and a person segmenter, both run on the render. It compares their outputs with the projected driving skeleton and the posed body-model silhouette, and adds a penetration check on the posed proxy geometry. This term targets the failures that Human Preference Score shows feature-statistics metrics miss, such as extra or fused limbs.
The appearance term uses known correspondences. It back-projects each rendered pixel to its canonical surface point through the posed body model. It then scores the rendered color by its normalized deviation from the colors that point showed across all training frames where it was visible. Points never visible in training are flagged as unobserved; they are reported separately rather than scored, because they are the measurement nullspace that HumanNeRF-style methods (HumanNeRF) must hallucinate. A part-conditioned patch statistic, computed per body part against that part's training patches, adds a check on texture sharpness. This follows the finding of Patch-Based Diffusion Models Beat Whole-Image Models that local patch statistics transfer across distribution shift better than whole-image statistics.
To validate the score, we borrow splitting directly. Within each multi-view capture, we rank frames by out-of-distribution distance, defined as the per-joint rotation distance to the nearest training pose; this follows the per-joint factorization in PoseVocab. We hold out the most extreme decile as a pose-stratified validation split, train avatars on the rest, and render the held-out poses from held-out cameras. On this split ground truth exists, so we can test whether the reference-free score tracks true error. We use full-reference structural error (keypoint and mask error against ground-truth detections) and human judgments, while also examining LPIPS on the same split. We then drive the same avatars with AIST++ dance poses, where no ground truth exists, and collect pairwise human preferences between renders from different methods.
As a downstream use, we take the role of the validation subset in Zero-Shot Self-Supervised Learning for MRI Reconstruction. The score on synthetic out-of-distribution driving poses selects regularization strength and the early-stopping point for an avatar method. We then test whether this improves human-judged out-of-distribution quality compared with selecting on held-out-frame PSNR.
Baselines
The naive reference point is the protocol now in use: PSNR, SSIM and LPIPS on held-out ZJU-MoCap frames. This is the protocol under which Locality Sensitive Avatars From Video reports over a 10% LPIPS and 20% FID improvement over Vid2Avatar and MonoHuman, and under which Representing Animatable Avatar via Factorized Neural Fields reports over 10% LPIPS and KID gains. On truly out-of-distribution poses it cannot be computed. There we use its natural extension, LPIPS against the nearest-pose training frame.
The strongest existing reference-free competitors are distribution metrics, FID and KID computed against the subject's training frames, as used in those two papers. The strongest learned judge in the file is HPS, which reaches 43.5% accuracy against 33.1% for CLIP on its own Human Preference Dataset (Human Preference Score). We run HPS with a fixed generic prompt describing a person. For agreement between any automatic metric and human preference on avatar novel-pose renders, we found no reported result on Lacuna. The human-agreement numbers for these baselines will therefore be measured here, all on the same render pool and the same preference pairs.
No metric is tuned on the test preference data. The two weights that combine our terms are fixed on the pose-stratified validation split, and baseline metrics receive the same opportunity, for example selecting the feature layer for LPIPS. Beating the baselines means higher pairwise agreement with humans on AIST++-driven renders than every baseline, with non-overlapping bootstrap intervals. It also means higher rank correlation with ground-truth structural error on the pose-stratified split.
The ablations are structure term alone, appearance term alone, and appearance with a global rather than per-point tolerance. The last one isolates the key ingredient: tolerance derived from each point's own observed variation, which is what is supposed to spare plausible wrinkles.
Why not just…?
Why not just use FID or KID against the subject's training frames? These metrics already avoid needing paired targets, and they are the current fallback. However, Human Preference Score documents that FID and IS fail to separate anatomically broken images from good ones. They are also set-level statistics, so they cannot localize which render or body part failed. The head-to-head agreement study will show whether this blindness carries over to avatars. If FID matches our score on human agreement, the structural machinery is not needed, and we would report that.
Why not just ask a learned preference model or a vision-language judge? This is a genuine threat, because preference models keep improving. HPS was trained on text-to-image aesthetics. It knows nothing about this subject's identity or clothing, and its authors report an "alignment tax" toward generic beauty, which suggests it would reward a smooth, plausible-looking person over a faithful one. We include HPS as a baseline. If it equals our score on human agreement, the conclusion is that subject-specific checking adds nothing, and the proposal reduces to a recommendation to use preference models.
Why not just hold out more extreme frames and keep using LPIPS? Our own validation split does exactly that, so this alternative is tested directly. LPIPS on that split is available as a comparison, and nothing prevents it from agreeing with humans. Two limits remain. Extreme held-out frames still come from the same capture session, so they are less out of distribution than cross-dataset driving. LPIPS also compares against one specific wrinkle realization, the single-reference problem raised by Yun Cao et al.. We will report how often LPIPS prefers a blurrier render that humans reject.
Why not just run a user study for every paper? Human studies are the ground truth here, but they are costly and cannot be reproduced across papers. We collect one study to validate the score so that later work does not need to.
Proposed Experiments
We train HumanNeRF, HumanNeRF-SE, Locality Sensitive Avatars, Factorized Neural Fields and AniGaussian, using public code where available. The data are ZJU-MoCap and PeopleSnapshot, with THuman4.0 and ActorsHQ for loose clothing. Each method is trained on the pose-stratified split, then rendered on held-out extreme poses and on AIST++ motions binned by out-of-distribution distance.
The first decisive check is calibration. On the held-out extreme poses, the score's rank correlation with ground-truth structural error must exceed that of LPIPS.
The second is human agreement on AIST++ renders, compared against every baseline and ablation.
The third asks whether method rankings flip between the standard protocol and the out-of-distribution bins. A flip would mean the field's leaderboard does not measure the property it claims. No flip, together with the score agreeing with LPIPS, would be a negative result worth publishing.
The fourth tests model selection. Hyperparameters chosen by our score should yield renders that humans prefer over those chosen by held-out PSNR.
Because the 2D keypoint detector may itself fail on contorted poses, we measure its error on ground-truth extreme frames. We report the structure term only within the range where the detector is reliable.
References
- Zero-Shot Self-Supervised Learning for MRI Reconstruction
- Equivariant Splitting: Self-supervised learning from incomplete data
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video
- Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems
- PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
- Locality Sensitive Avatars From Video
- Representing Animatable Avatar via Factorized Neural Fields
- Unsupervised Diverse Colorization via Generative Adversarial Networks