Is Body-Shape Bias Partly a Resizing Artifact? Treating Crop Warping as a Camera in 3D Human Mesh Recovery

Systems that reconstruct a full 3D body surface from a single photo often get people's body shape wrong. They are good at recovering where the joints are, but they tend to pull every body toward an "average" build. A heavyweight shot-putter comes out slimmer and a gymnast comes out bulkier. This note proposes a specific, testable explanation for part of that failure.

To a camera, a person's girth is literally a ratio between horizontal and vertical extent in the image. Most pipelines resize the person's bounding box into a fixed square before the neural network sees it. Many also stretch images during training augmentation. Either step changes that ratio by an amount the network is never told. A network that sees the same body arbitrarily squashed and stretched during training can learn the wrong lesson: that the width-to-height ratio of a silhouette is unreliable. The safest bet is then to predict an average girth. If this is right, much of the "average body" bias is learned indifference to a corrupted signal, not a shortage of diverse training bodies.

The fix follows directly. A non-uniform resize is mathematically the same as changing the camera's focal length differently along the two image axes. So the network should keep the person at native aspect ratio, and it should also be told, pixel by pixel, which camera produced the pixels it is looking at.

Core Claim

We claim that anisotropic crop warping is a measurable and correctable cause of shape regression-to-the-mean in single-image human mesh recovery. Anisotropic warping means unequal horizontal and vertical scaling, whether from squaring a non-square box or from aspect-ratio augmentation. The correction has two parts. First, the encoder takes variable-aspect-ratio patch tokens instead of a square grid. Second, every patch carries a dense map of the viewing ray it corresponds to, which states the effective per-axis focal lengths after preprocessing.

The geometry is simple. If a crop is rescaled by factors $s_x$ horizontally and $s_y$ vertically, the effective intrinsics change from $(f_x, f_y)$ to $(s_x f_x, s_y f_y)$, where $f_x$ and $f_y$ are the camera's focal lengths in pixels. Once these are known, apparent width relative to height maps back to metric girth. Without them, girth is unidentifiable from the image alone whenever $s_x/s_y$ varies.

The claim is new in two ways. It separates the two candidate causes of shape bias, warping versus dataset skew, which existing work does not do. And it treats preprocessing as part of the camera model rather than as a nuisance to pad away.

Where This Sits in Prior Work

Work that motivates the proposal. Shape of You diagnosed the "average athletic body" bias as entanglement between silhouette and stance. It fixed the bias with DensePose correspondence losses, losses computed in a pose-invariant T-pose space, and about 8 seconds of test-time refinement per image. It reduces the error metric PVE-T-SC on SSP-3D from 19.2mm (SHAPY) to 15.8mm. It does not ask whether its square-crop input contributes to the bias. We reuse its T-pose loss and ask exactly that question.

Mask as Supervision aligns a learned "physique mask" with 2D silhouettes, using geodesic weighting from the body centroid. That alignment is only meaningful if mask proportions are undistorted, and the paper uses fixed-size crops. Its TikTok dance setting, which consists of vertical phone video, is the natural place where our effect should be largest.

Truth or Square reports that non-square aspect ratios bias how positional encodings recall location. This is independent evidence that the encoder's notion of space is itself sensitive to aspect ratio, not just its input pixels.

Work that supplies the method. NaViT processes images at native aspect ratio. It packs patches from several images into one sequence, uses masked attention so images do not attend to each other, and uses factorized x/y positional embeddings. It matches standard ViT accuracy with about four times less compute, and it noted fairness gains on human attributes. However, it never predicts 3D bodies. We take its packing and positional scheme directly.

Thinking with Camera (Puffin) shows that dense, pixel-wise camera maps help models reason about perspective, reaching a median pitch error of 1.08° on MegaDepth. It works only on scenes, not bodies. Our ray map is the same idea, repurposed so that it also encodes preprocessing-induced anisotropy.

3D Human Mesh Estimation from Virtual Markers is the efficient feedforward reference. It predicts 64 surface markers and then reconstructs the full mesh, reaching 44.7mm MPVE on SURREAL at 22.1 GFLOPs.

What we deliberately do not adopt. We do not use LLaVA-UHD's slicing of an image into tiles. A single person crop is small, and splitting it into tiles would create exactly the boundary discontinuities across a torso that we want to avoid.

Research Program

The work has three parts, and the first is designed to be able to kill the second.

A diagnostic: the stretch-response test. We take existing mesh regressors and measure how predicted girth responds to controlled horizontal stretching of the input. Girth here means waist and chest circumference, computed from the predicted T-pose mesh. We also measure how predicted girth tracks true girth across subjects, through the slope $g$ of a linear fit of predicted circumference on ground-truth circumference. A value of $g = 1$ means no regression to the mean; $g < 1$ means bodies are pulled toward average.

On SURREAL we can additionally render the same body under controlled preprocessing, because ground-truth shape is known there. We then train identical models with two recipes. One recipe applies anisotropic warping, by squashing or stretching boxes to a square and adding aspect jitter. The other is warp-free, either padded or native-aspect. Both use matched shape distributions.

Our hypothesis predicts that the warped recipe yields a lower $g$ and a near-zero response to stretching. If $g$ is the same under both recipes, warping is not the culprit, and we report that as the main result.

The model. A NaViT-style encoder takes the person crop at native aspect ratio. Each patch embedding is concatenated with a small encoding of its ray map, meaning the normalized ray direction at the patch center computed from $(s_x f_x, s_y f_y)$ and the principal point. When true intrinsics are unknown, as on in-the-wild images, we use a nominal field of view but still encode $s_x/s_y$ exactly, because that ratio is always known from preprocessing.

The output head regresses SMPL pose, shape and camera parameters, trained with the T-pose vertex loss from Shape of You. During training we deliberately apply random anisotropic warps while updating the ray map to match. The model therefore sees distortion it can always explain, which is the opposite of the uninformed jitter we hypothesize causes the bias.

Analysis. We measure shape gain $g$ separately per BMI band, and stretch-invariance (whether shape stays fixed when the image is warped and the ray map is updated). We also characterize behavior on extreme aspect ratios: tall standing people at 1:4, and wide lying or reaching poses.

Baselines

The naive reference point is a standard square-warp regressor, where the bounding box is resized directly to 224×224, using the same ViT size, training data and schedule as our model. This is the pipeline our hypothesis indicts. It sets the floor for $g$ and for PVE-T-SC.

The strongest geometric competitor is square context-expansion, or padding. The shorter side of the box is extended with real background or zeros until the crop is square, so there is no anisotropic distortion at all. This is the baseline most likely to erase our advantage, because it removes the distortion we target without any architectural change. For fairness we run it at two budgets. At matched token count, it has lower effective resolution on the person. At matched person resolution, it has more tokens and more compute. We report both FLOPs and accuracy, since NaViT's evidence is about compute, not accuracy.

The strongest published shape method is Shape of You. It reports PVE-T-SC of 15.8mm and mIoU of 0.76 on SSP-3D, but only with roughly 8 seconds of test-time refinement per image. We compare our feedforward model against its reported numbers and against our reimplementation of its feedforward stage, without refinement, using our backbone. We also add refinement on top of our model to test whether the two gains stack.

For efficient dense reconstruction we compare against Virtual Markers on SURREAL, where it reports 44.7mm MPVE. For 3DPW and Human3.6M mesh error, no reported result was found on Lacuna for Virtual Markers. We will run its released configuration ourselves.

The ablation that isolates the key ingredient keeps native-aspect tokens but drops the ray map. A second ablation keeps the ray map but gives only the scalar $s_x/s_y$ ratio instead of a dense field.

Beating the field means three things. Our model must improve on square-warp in both PVE-T-SC and $g$. It must match or beat padding at equal FLOPs. And it must approach Shape of You's 15.8mm without test-time optimization.

Why not just…?

Why not just pad the box to a square? Padding removes distortion, and if the bias comes only from warping, padding may recover most of the gain. This is the real threat. The diagnostic includes padding as one of the warp-free recipes precisely so that this outcome is visible. If padding closes the gap, the honest conclusion is that cheap padding fixes a real bias, and the remaining contribution of native tokens is efficiency on tall crops, where NaViT documents large compute savings. What padding cannot do is use aspect augmentation safely, or exploit known intrinsics. The ray-map ablation measures whether either of those matters.

Why not just train on more diverse bodies? Dataset skew is the leading explanation in Shape of You, and we do not dispute it. The SURREAL controlled experiment holds the shape distribution fixed while varying only preprocessing. This separates the two causes, and it can also show that warping contributes nothing.

Why not just rely on test-time refinement? Refinement costs about 8 seconds per image. Shape of You also notes that it shrink-wraps the mesh onto corrupted correspondences, for example under loose clothing. We test whether refinement started from our model converges faster and lands better than refinement started from square-warp. If it does not, the feedforward gain is not compounding.

Why not just pass the scale factors as a scalar? This is our second ablation. If a single $s_x/s_y$ token suffices, we adopt it. A dense map is only justified if it helps on wide-field-of-view crops away from the image center, where perspective varies across the body.

Proposed Experiments

We train all models on the same mixture, consisting of SURREAL and SMPL fits from AMASS rendered into images. We use an identical ViT size and schedule, and matched tuning budgets for all variants.

The first decisive check is the SURREAL controlled diagnostic. Under a fixed shape distribution, does warp-trained $g$ fall measurably below warp-free $g$? This experiment determines whether the project's premise holds.

The second check is SSP-3D, using PVE-T-SC and mIoU broken down by BMI band. Our prediction is that the gains concentrate at the high and low extremes. A uniform improvement would point to added capacity rather than to fixed geometry.

The third check is the stretch-invariance test on held-out images. With the ray map updated, predicted shape should stay fixed under warps from 0.8× to 1.25×. Square-warp models should either drift with the warp or stay rigidly at the mean.

Finally, we report 3DPW and Human3.6M joint error (MPJPE) to confirm that pose accuracy does not degrade. We also report qualitative and silhouette-IoU results on TikTok-style vertical video, the regime with the most extreme aspect ratios. No reported numbers were found on Lacuna for 3DPW or TikTok against our baselines, so all of these comparisons will be run in-house under the protocol above.