Most real-image training data for 3D human pose and shape estimation does not come with measured 3D labels. It comes with "pseudo ground truth": an optimizer fits a parametric body model to detected 2D keypoints, and the fitted result becomes the training label. Recent work rebuilt this fitting step around a full pinhole camera so that close-range, wide-angle images stop producing warped bodies. Two things have not been checked. The first is whether these fitters actually reach the optimum of their own objective. The second is whether that matters most in exactly the close-range regime they were built for. That regime is badly conditioned, because the focal length and the subject's distance from the camera can trade off against each other while leaving the image almost unchanged. Iterative solvers that stop early along such a nearly flat direction tend to return bodies that are still partly distorted. Nothing flags the failure.
This note proposes to measure that hidden error and follow it downstream. We will build a fitter based on the semismooth Newton method, a second-order solver for objectives with kinks and hard constraints that converges very fast near the solution. Its job is to produce a reference fit with a verifiable optimality certificate, and we will use it to split pseudo-ground-truth error into a solver part and a modelling part on synthetic images where the true answer is known. We then ask the question that matters for the synthetic-data literature. When people report how many synthetic images are "worth" one real image, how much of that exchange rate comes from noise in the real labels rather than from realism in the synthetic ones?
Core Claim
Our central claim is that solver inexactness is an unmeasured variable in perspective-aware 3D human labels. We expect it to grow as the camera gets closer to the subject, and to bias published estimates of how replaceable real data is.
The claim has three testable parts. First, on synthetic close-range images with known camera and body, standard first-order or quasi-Newton fitting leaves a measurable residual at the optimality conditions. That residual correlates with error in recovered depth and focal length, and the correlation strengthens as camera distance falls from 10 m toward 0.5 m. Second, solving the same objective to a certified tolerance removes most of this solver error. Third, if the real-image labels are refit this way, the number of synthetic frames needed to match a given amount of real data changes measurably.
The third part is the new scientific contribution. Equivalency ratios have so far been measured where real labels are human annotations, not optimizer outputs. In 3D human estimation, a "real" frame carries the fitter's errors with it. A synthetic-versus-real exchange rate is therefore partly a measurement of the fitter.
We do not claim that semismooth Newton is the only route to an exact fit. Damped Gauss–Newton methods may get there too, and they are a first-class competitor below. The solver is the instrument; the finding about label quality and exchange rates is the point.
Background and Positioning
The perspective line of work supplies the motivation. SPEC showed that estimating camera pitch, roll and field of view and feeding them into a perspective-aware SMPLify-X reduces world-frame error. Its camera estimator leans on background lines and degrades on tight portraits. Zolly treats distortion as a cue rather than noise and infers the subject's distance and focal length from the body itself, trained on the synthetic PDHuman set spanning 0.5–10 m. It admits failures on atypical body shapes, which is the same distance–focal–shape ambiguity we target. CameraHMR is the direct starting point. It predicts field of view from people (HumanFoV), predicts 138 dense surface keypoints trained on BEDLAM and AGORA, fits SMPL under a full perspective camera (CamSMPLify), and retrains a regressor on the refreshed labels.
All three treat the fitting optimizer as a fixed utility. None reports convergence diagnostics, and none separates errors caused by the camera estimate from errors caused by incomplete optimization. Our proposal adds exactly that separation. Classical results ground why it should matter at close range. Kissos et al. and Mori et al. show that weak-perspective assumptions corrupt joint orientation and depth when the subject's own depth extent is not small relative to its distance from the camera. Prados et al. report that switching to a pinhole model cut facial reconstruction error from about 15% to 3%.
The synthetic-data line supplies the evaluation lens. Chang et al. defined an equivalency ratio and a substitutability threshold for multi-person tracking with MOTRv2. They report roughly 30 synthetic frames per real frame, and that 60–80% of real data can be replaced after synthetic pre-training without loss. Their real labels were human-annotated boxes. BEDLAM showed that synthetic-only training reaches 43.0 mm PA-MPJPE on 3DPW, and that 5% of BEDLAM beats larger, less realistic synthetic sets. Neither asks how the quality of real labels shifts the measured value of synthetic data. That question only arises when real labels are themselves produced by an optimizer.
The optimization line supplies the method. The defining move of semismooth Newton is replacing an undefined Hessian at a kink with a generalized one. For soft-thresholding, SSNAL shows this generalized Jacobian is a 0/1 diagonal, so each Newton step only involves the currently active coordinates. ProxQP and ProxDDP wrap such steps in a primal-dual augmented Lagrangian, a scheme that adds a quadratic penalty and multiplier estimates for the constraints so the Newton steps stay well-behaved far from the solution. ProxDDP does this for nonlinear trajectories whose stage-wise structure resembles a kinematic chain.
Reported speedups in this literature, such as 330× over ADMM on Lasso, come from convex, sparse problems and should not be assumed to carry over to nonconvex body fitting. Won studies a "matrix perspective function." That is a convex-analysis object unrelated to camera projection, and we do not build on it beyond noting that the shared word is a coincidence.
Research Program
The fitting objective. We take the CamSMPLify objective as given and make its nonsmooth pieces explicit rather than smoothed. The pieces are:
- a robust reprojection loss on the dense keypoints under full perspective;
- a body-shape and pose prior;
- hard box constraints on joint angles;
- a positivity constraint keeping every vertex in front of the camera by a margin;
- optionally, the focal length as a variable confined to an interval around the HumanFoV estimate.
For the robust loss we use the Huber function, which is quadratic for small residuals and linear for large ones. Its generalized Hessian is again a 0/1 selection: inlier keypoints contribute curvature, outliers contribute none. The joint-angle constraints are handled by projection, whose generalized Jacobian is also a 0/1 selection over inactive versus active angles. Both inherit the active-set sparsity that makes semismooth Newton steps cheap.
The solver. The perspective division and the body kinematics make the problem nonconvex. Following ProxQP's observation that the proximal regularization must dominate the most negative curvature, we use a proximal augmented Lagrangian outer loop with an adaptive proximal weight. A semismooth Newton inner solver exploits the kinematic-tree structure in the style of ProxDDP's stage-wise recursion.
The optimality certificate. The solver outputs a stationarity residual: how far the current fit is from satisfying the first-order optimality conditions, including constraint and multiplier terms. We apply the same residual to every competing solver, so all are judged by one yardstick and not by the objective value alone.
The synthetic audit. We render held-out scenes in the PDHuman and BEDLAM styles with known body, camera and distance, stratified by distance from 0.5 to 10 m. Each scene is fitted under four conditions:
- ground-truth keypoints with ground-truth intrinsics, which isolates pure solver error;
- predicted dense keypoints with ground-truth intrinsics;
- ground-truth keypoints with HumanFoV intrinsics;
- the full pipeline.
For each condition we record vertex error, depth error, focal error and the stationarity residual, per solver and per distance bin.
Real-data relabeling and exchange rates. We refit the CameraHMR real training pool (4D-Humans-style images from COCO, MPII, AI Challenger and InstaVariety) with each solver, then retrain the same regressor on each label set. We then measure substitutability curves by training on mixtures that replace a fraction of real frames with BEDLAM frames. We define the exchange rate at a given real-data budget as the number of synthetic frames that, added to a reduced real set, restores the downstream error of the full real set, following Chang et al.'s definition. We report the curves once with original labels and once with exactly solved labels.
Baselines
The naive reference is weak-perspective SMPLify pseudo ground truth with a default camera, as used in older real-image training pools. It anchors how much any perspective-aware fitting helps at all.
The strongest existing method is CameraHMR's CamSMPLify with its published optimizer, plus the CameraHMR regressor trained on its labels. It is the current perspective-aware labelling pipeline, and our relabeling is a drop-in replacement for exactly this stage. The file reports CameraHMR at 31.8 mm PA-MPJPE on SPEC-SYN, against 55.8 mm for CLIFF and 51.8 mm for TokenHMR. On 3DPW the file gives no CameraHMR number. The strongest reported 3DPW figures are Zolly's 39.8 mm PA-MPJPE and the BEDLAM-only model's 43.0 mm. For SPEC-MTP, no reported result found on Lacuna for CameraHMR, SPEC or Zolly. We will rerun all three under our protocol rather than quote them.
The most dangerous competitor at the solver level is a damped Gauss–Newton fitter (Levenberg–Marquardt with joint-angle clipping and a smoothed robust loss), the standard workhorse for vision least-squares problems. It gets second-order behaviour without semismooth machinery.
Fairness rules apply to every solver. All of them, including L-BFGS and Adam as published in CamSMPLify, receive identical objectives, initializations from the same regressor, the same keypoints and intrinsics, and either a matched wall-clock budget per image on the same GPU or a matched target stationarity residual, reported both ways. Hyperparameters for each solver are tuned on the same small synthetic validation split with the same trial budget. Downstream regressors share architecture, schedule and seed count, and we report means over at least three seeds.
Two ablations isolate the key ingredient. The first keeps our solver but replaces the hard joint-angle constraints and the Huber loss with smooth surrogates, testing whether exact handling of the kinks matters. The second runs the first-order solver to the same certified residual our solver reaches, however long that takes, testing whether the label improvement comes from precision itself or from where the solver lands.
Beating CamSMPLify means three things. On the synthetic audit, lower vertex and depth error in the 0.5–2 m bins at matched wall clock. Downstream, a statistically reliable reduction in PA-MPJPE or MPJPE on SPEC-MTP and EMDB when the regressor is trained on our labels. Without regression on 3DPW.
Why not just…?
Why not just use Levenberg–Marquardt, the standard vision solver? This is a real threat, and we treat it as a co-equal candidate, not a strawman. Its weakness in principle is that it needs smooth surrogates for the joint limits and the robust loss, and clipping after a step can break its convergence near active constraints. Whether that matters in practice is exactly what the audit measures, on two axes: stationarity residual and close-range error at matched time. If Levenberg–Marquardt matches our solver, the solver-specific contribution shrinks to a negative result. The central findings about hidden solver error and confounded exchange rates still stand, using whichever exact solver wins.
Why not just run L-BFGS or Adam for more iterations? If longer runs close the gap, the conclusion is "CamSMPLify is under-iterated," which is still worth knowing. The precision-matched ablation tests this directly. Our prediction is that first-order methods slow sharply along the nearly flat direction where focal length and distance trade off, so that matching the certified residual costs many times more compute across hundreds of thousands of images. The audit's iterations-to-tolerance curves per distance bin will confirm or refute this.
Why not just skip real labels and train on synthetic data only? BEDLAM-only training reaches 43.0 mm on 3DPW, while Zolly reports 39.8 mm. Synthetic data alone is competitive but not clearly sufficient, and BEDLAM's barefoot avatars without object contact leave known gaps. The zero-real point of our substitutability curve tests this alternative directly. If synthetic-only training matches the relabeled mixtures on SPEC-MTP, real-label quality is moot, and we will report that.
Why not just improve the camera estimate instead of the solver? Camera error and solver error are different sources, and the four-condition audit separates them. If the ground-truth-intrinsics condition shows negligible solver error at every distance, then the solver story is wrong and effort should go to field-of-view estimation. That outcome kills the proposal cheaply and early.
Decisive Checks
The first check is the kill test and costs only rendering and fitting. On the synthetic audit with ground-truth keypoints and intrinsics, does the published optimizer leave a stationarity residual and a depth or focal error that is both non-negligible next to the error from predicted keypoints and increasing as distance falls? If not, we stop.
The second check concerns the labels themselves. On a subset of real images that also has independent ground truth, from SPEC-MTP and EMDB, do exactly solved labels sit closer to that ground truth than CamSMPLify labels, especially for subjects close to the camera?
The third check is the downstream payoff. Does a regressor trained on relabeled data improve on SPEC-MTP and EMDB, and on the close-range PDHuman test split, while holding 3DPW, RICH and SSP-3D? SSP-3D additionally probes whether atypical body shapes are still mistaken for perspective distortion.
The fourth check answers the scientific question. Do the substitutability curve and the exchange rate against BEDLAM shift when only the real labels change? We predict a shift in the direction of real frames becoming more valuable, meaning a higher synthetic-per-real ratio and a lower safe replacement fraction, because cleaner real labels carry more information. A shift of either sign would establish that exchange rates in label-optimized domains cannot be quoted without reporting how the real labels were solved. No shift would show that published exchange rates are robust to fitter quality. Either outcome is a result the synthetic-data literature currently lacks.
References
- SPEC: Seeing People in the Wild with an Estimated Camera
- Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction
- CameraHMR: Aligning People with Perspective
- Beyond Weak Perspective for Monocular 3D Human Pose Estimation
- Estimating Human Body Configurations Using Shape Context Matching
- "Perspective Shape from Shading" and Viscosity Solutions
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion
- A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems
- ProxQP: an Efficient and Versatile Quadratic Programming Solver for Real-Time Robotics Applications and Beyond
- ProxDDP: Proximal Constrained Trajectory Optimization
- Proximity Operator of the Matrix Perspective Function and its Applications