One Scale per Person: Gauge-Free Mesh-Anchored Gaussian Avatars from Monocular Video

A single camera cannot tell a tall person far away from a short person close by. In every frame, scaling the whole scene by any factor leaves the image unchanged. Avatar systems that attach 3D Gaussians (small soft colored blobs, rendered by fast differentiable splatting) to a body mesh such as SMPL quietly assume this problem is already solved. They take each frame's body size and distance from an external pose estimator and then fit appearance on top. Those per-frame estimates are not consistent with one another: the estimator can make the body slightly nearer in one frame and slightly larger in the next. Because the mesh is held fixed, the Gaussians attached to it absorb the mismatch. They stretch along the camera's line of sight, the body's pose foreshortens, and splats grow into needles. The pictures look fine from the training camera and break from any other viewpoint.

This note proposes treating scale as exactly what the projection geometry says it is: one unknown number per subject. Reconstruction runs in a scale-free canonical space where image evidence cannot see that number. A separate, explicitly uncertain estimate then fuses independent metric cues to set it: a metric depth model, a statistical prior on human body size, and ground contact. A second ingredient handles the depth ambiguity that remains within each frame. Each Gaussian's departure from the mesh is regularized according to how much the training views actually constrain it along each direction, so offsets that only the line of sight could have produced are pulled back toward the mesh. The claim is that removing the arbitrary per-frame scale choice improves novel-view geometry and appearance. The test is whether the gain survives after the baselines are given oracle global rescaling, not only whether metric height comes out right.

Core Claim

Under a pinhole camera with fixed focal length, scaling the body, its offsets and its root trajectory together by a factor $s>0$ produces the same pixels. For the photometric loss, $s$ is therefore a pure gauge, meaning a free parameter the data cannot pin down. Current mesh-anchored pipelines such as GauHuman and EVA assume calibrated cameras and inherit SMPL size and root translation from an estimator. They therefore fix this gauge differently in every frame. DGG-HMR documents the same depth–size cheating in mesh recovery: networks shrink or enlarge bodies to satisfy 2D reprojection. We hypothesize that in avatar fitting the leftover inconsistency does not vanish but leaks into the Gaussian layer as ray-aligned distortion. Two things should reduce that distortion: fixing one shared scale per subject, and weighting offset regularization by per-direction observability. The improvement should show up in similarity-aligned novel-view metrics, beyond what post-hoc metric rescaling can provide.

Research Program

Representation. Each Gaussian $k$ is bound to a triangle of a scale-normalized body mesh, whose height is fixed to one unit. Its position at frame $t$ is

$$\mu_k(t) = s\,\Big[\mathrm{LBS}_t\big(\bar v_k + \ell_k\,F_k\,\delta_k\big) + \bar r_t\Big].$$

Here $\bar v_k$ is the anchor point on the canonical mesh. $\ell_k$ is the local triangle edge length, so offsets are measured in units of local surface size. $F_k$ is the triangle's local frame, and $\delta_k$ is the learned residual offset. $\mathrm{LBS}_t$ is linear blend skinning, which deforms the mesh by the frame's skeletal pose. $\bar r_t$ is the scale-free root translation. Gaussian scales are expressed in the same units.

Body proportions, pose, per-frame root depth along the viewing ray, and $\delta_k$ are all optimized with $s$ fixed to 1. Rendering is exactly invariant to $s$, so the photometric optimization cannot trade size against distance. All frames share one body size, while each frame keeps its own depth. This differs from GauHuman and EVA, which accept external per-frame translations. It also differs from Mode-GS, which optimizes a separate scale factor per view for static point anchors. In our setting, per-view scales are exactly the freedom that lets errors leak.

Metric scale as a fused estimate. $s$ is estimated from three cues, each turned into a likelihood over $s$, and the likelihoods are combined. The first cue is Metric3D v2 depth on person pixels. This model normalizes focal length into a canonical camera to predict metric depth, so it is only trustworthy when focal length is known. The second is an anthropometric height prior evaluated on the recovered body proportions. It follows the statistical-body reasoning of Barron et al., and Fei et al. show it is biased for children and non-upright postures. The third is foot–ground contact when a ground estimate exists, the cue that Crowd4D exploits at multi-hour cost. Each cue's variance is calibrated on held-out data. The output is a posterior over $s$, not a point estimate, so downstream users can see when scale is poorly determined.

Observability-weighted offset regularization. Even with a shared scale, each frame constrains a Gaussian mostly perpendicular to the viewing ray. USplat4D showed that this anisotropy matters for unanchored dynamic Gaussians, using closed-form uncertainty derived from photometric convergence. We carry the idea into mesh-local coordinates. For each Gaussian we accumulate an approximate information matrix $H_k = \sum_t J_{k,t}^\top J_{k,t}$, where $J_{k,t}$ is the Jacobian of its projected position in frame $t$ with respect to $\delta_k$. We then regularize with

$$\mathcal{R} = \sum_k \delta_k^\top \big(\lambda\, (H_k + \epsilon I)^{-1}\big)\,\delta_k ,$$

where $\lambda$ sets the strength and $\epsilon$ prevents division by zero. Directions seen from many angles barely pay the penalty, so a rotating subject keeps full detail. Directions observed only along the line of sight, such as a subject who always faces the camera, are pulled toward the mesh. In practice $H_k$ is recomputed every few hundred iterations, using a $3\times3$ block per Gaussian.

Baselines

The naive reference is GauHuman run exactly as published, with per-frame SMPL and root translation from an off-the-shelf estimator and the dataset's focal length. It is the cheapest strong mesh-anchored avatar method. The file reports 33.45 PSNR on MonoCap with calibrated inputs (GauHuman). We rerun it with the same estimated inputs our method receives, so the comparison isolates how it handles scale and not the quality of its inputs.

The strongest competitor is a pipeline built for exactly this question: per-frame metric body meshes from DGG-HMR, which reports 61.1mm MPJPE on 3DPW, fed into EVA's SMPL-X Gaussian fitting. It resolves metric depth with a depth foundation model and then renders with a strong anchored representation, so it is the obvious way to get "metric avatars" without new machinery. We also give it temporally smoothed root trajectories, because jitter is the cheapest thing to fix. All methods share the same estimated poses, the same focal handling, the same number of optimization steps, and a matched hyperparameter search of equal trials on a validation subject.

MoDGS represents the template-free alternative. It is a deformable Gaussian model with an ordinal depth loss and reports 22.64 PSNR on DyNeRF. Its affine ambiguity makes it a natural foil for the metric question. It is run on the same human sequences.

Two ablations isolate the ingredients. The first keeps the scale-free gauge but replaces the observability-weighted penalty with an isotropic one. The second keeps the penalty but returns to per-frame external scale. Beating the strongest pipeline means lower error on similarity-aligned novel-view metrics. Metric accuracy alone does not count.

Why not just…?

Why not just fit metric SMPL with DGG-HMR and then run a standard Gaussian avatar on top? This is the real threat. If DGG-HMR's per-frame depths are consistent enough, our gauge gives no extra benefit. DGG-HMR estimates each frame independently and stops gradients from the mesh into the depth branch, so nothing enforces one body size across a video. Whether that inconsistency matters is exactly what the injected-error experiment below measures. If the strongest pipeline matches us after alignment, we will report that the contribution reduces to scale estimation.

Why not just reconstruct up to scale and rescale afterward using an assumed height? A single post-hoc rescale fixes metric placement but cannot undo pose foreshortening or needle splats created during fitting. We test this directly by giving every baseline oracle similarity alignment to ground truth. If our aligned metrics are no better, the reparameterization contributes nothing beyond what Fei et al.-style calibration already provides.

Why not just supervise unanchored dynamic Gaussians with metric depth, possibly with per-view scale calibration as in Mode-GS? Unanchored Gaussians degrade when viewpoints change substantially. USplat4D reports that they collapse at large view shifts, and Mode-GS was evaluated only on static scenes. Per-view scale factors reintroduce the per-frame freedom we remove. The MoDGS comparison and a Mode-GS-style per-view-scale variant of our own model show which effect dominates.

Proposed Experiments

The main testbeds are ZJU-MoCap and DNA-Rendering, both multi-view with calibrated metric cameras. Methods train on one camera's video using estimated poses and are evaluated on held-out cameras. Each held-out view is scored twice. The first score uses the true metric camera, which tests scale. The second applies a similarity alignment to ground truth first, which tests shape and appearance. The file has no reported result on Lacuna for monocular-input ZJU-MoCap or DNA-Rendering under estimated poses, so all numbers there are produced by our reruns.

Loose clothing is the stress case. Large offsets give ray-aligned leakage the most room, so we use 4D-Dress, where Secondary Motion-Aware Gaussian Avatars report 24.74–29.98 PSNR. LIFe-GoM acknowledges that Gaussians-on-mesh fail on such garments. We also use 3DPW for metric root depth and height error in the wild.

The metrics are PSNR, SSIM and LPIPS in both scoring modes, recovered height error in centimeters, root-depth error, and Chamfer distance to scans where they exist. We also report a needle-artifact rate: the fraction of Gaussians whose largest axis is more than ten times their smallest, measured along the ray direction.

The decisive check is a controlled corruption sweep. Starting from ground-truth calibration, we perturb per-frame root depth by independent errors of 5–30% and focal length by ±20%. We then plot aligned novel-view LPIPS and needle rate against corruption level for each method. The proposal predicts that GauHuman and the DGG-HMR→EVA pipeline degrade steadily, while ours stays flat, because per-frame depth error under a shared scale can only move the body along rays, and the information-weighted penalty discounts exactly that direction.

The ablations separate the two ingredients: the gauge should dominate under depth jitter, and the penalty should dominate on front-facing sequences. A final check is calibration of the scale posterior. Its 90% intervals on recovered height should contain the truth about 90% of the time, including on child subjects, where the anthropometric cue alone is known to fail.