Virtual-reality headsets see only three points on the body: the head and the two hand controllers. From those three points a tracker has to produce a full skeleton, and the answer depends on the user's body proportions. A tall person with short legs and a short person with long legs can produce nearly the same headset and controller trajectories. The most robust recent tracker, InPose, generalizes across body sizes, but only because it is handed the user's exact bone lengths at inference time. Its authors name automatic estimation "during the first few seconds of user calibration" as future work. We propose to solve that calibration step as a cold-start optimization problem. Each trial value of the bone lengths is scored by running the tracker and checking how well the result explains the sensor stream. That check is expensive, so we can afford only a few dozen of them. The first few must be chosen well, and the choice should also tell the system which proportions the data cannot yet pin down.
The core idea is short. We treat the user's 8 or so bone-length parameters as the unknowns of a black-box objective and borrow two techniques from Bayesian optimization (BO), the family of methods that finds the optimum of an expensive function by fitting a probabilistic model to a few evaluations and choosing the next query from it. The first is model-aware initialization, which spends the first evaluations on learning which parameters the objective is sensitive to. The second is a population prior that guides early queries and then fades away. Where the objective stays flat along some direction, for example how standing height splits between torso and legs, the same uncertainty model tells the system which short prompted motion would resolve it.
Core Claim
Our claim is that the first calibration evaluations and seconds should be allocated by information about the user's own proportions, not by a fixed script or a space-filling design. Calibrated this way, a scale-free inverse solver like InPose should come within a small margin of its oracle accuracy, the accuracy it reaches when given true bone lengths. It should do so with far fewer solver evaluations and fewer seconds of user motion than Sobol-initialized BO, a fixed calibration routine, or a learned regressor from passive data. The gains should be largest exactly where current trackers fail, on unusual proportions and on the lower body.
Problem Setup
Let $\theta \in \mathbb{R}^d$ be the user's bone lengths, with $d \approx 8$ after imposing left/right symmetry. Let $y_{1:T}$ be the headset and controller stream from the first $T$ seconds. We define the calibration objective as the misfit left over after the inverse solver reconstructs motion under candidate lengths $\theta$:
$$L(\theta) = \frac{1}{T}\sum_{t=1}^{T} \big\| \mathrm{FK}_\theta(\hat r_t(\theta)) - y_t \big\|^2 + \lambda \, \| \theta - \theta_{\mathrm{pop}}(y) \|^2_{\Sigma^{-1}} .$$
The first term measures how well the solver explains the sensors. Here $\hat r_t(\theta)$ are the joint rotations InPose recovers at frame $t$ given lengths $\theta$, and $\mathrm{FK}_\theta$ is forward kinematics, which places the tracked endpoints from rotations and lengths. The second term is a regularizer. It pulls $\theta$ toward $\theta_{\mathrm{pop}}(y)$, the mean of the bone-length distribution across AMASS subjects conditioned on cheap measurements such as standing head height and arm span. $\Sigma$ is the covariance of that distribution, and $\lambda$ weights the term.
Each evaluation of $L$ runs a stochastic diffusion solver over a window of frames. InPose reports roughly 229 samples per second on an RTX Titan, so a calibration budget is tens of evaluations, not thousands. That is the classic BO regime. The objective is also partly unidentifiable. With passive standing and arm motion, $L$ is nearly flat along the torso-versus-leg direction. That flatness is the sparse-sensor ambiguity, here seen from the calibration side.
Research Program
The method has three parts, each taken from a specific line of prior work.
The surrogate is a Gaussian process (GP), a probabilistic regression model, fitted over $\theta$ to noisy evaluations of $L$. It has one lengthscale per bone-length dimension, so a long fitted lengthscale means the objective is insensitive to that dimension.
The initial batch of evaluations is chosen with the criterion from Informed Initialization for Bayesian Optimization and Active Learning (HIPE). HIPE combines two scores. The first rewards queries that reduce uncertainty about predictions across the space. The second, called BALD, rewards queries that sharpen the model's estimate of its own lengthscales. On a 40-dimensional SVM tuning task HIPE learned the relevant lengthscales about half an order of magnitude more accurately than Sobol sequences within the first batch. HIPE supplies the method here, but it has only been tested on synthetic functions and ML hyperparameter tuning. Our setting differs in that low sensitivity in a dimension is not a nuisance to be ignored. It is a diagnosis that the data cannot determine that bone length.
The GP prior mean is the population-conditioned misfit, mixed in with a weight that decays, following Dynamic Mean Decay (DynMeanBO). DynMeanBO blends an expert prior into the GP mean with a weight that shrinks each iteration, which keeps convergence guarantees even when the prior is wrong. We choose this over space warping, which reshapes the search space around the prior. Space warping has a zero-probability trap: a region the prior excludes can never be found. That rules it out for exactly the unusual users we care about, so it constrains our design rather than supplying it.
The active step closes the loop. After the initial batch, a dimension may keep a long fitted lengthscale and wide posterior uncertainty. For each such dimension, the system ranks a small library of 3–5 second prompted motions, such as a squat, a forward step, kneeling to touch the floor, or sitting. We pre-compute on AMASS how much each motion would reduce bone-length posterior uncertainty, using simulated three-point streams from subjects whose proportions match the current posterior. The highest-ranked motion is requested, and BO continues on the extended stream.
This turns the passive heuristic in the Transformer IMU Calibrator into an active one. That method only calibrates when natural motion happens to vary enough in rotation, and it simply does not run otherwise. It also differs from the learned initialization network of PIP, which assumes the user's height and weight are already known.
The deliverables are the calibration procedure, an analysis of which bone-length directions are identifiable from passive versus prompted three-point data, and a prompt library ranked by information gain.
Baselines
The naive reference is InPose run with the AMASS mean skeleton and no calibration. It shows how much error comes from not knowing proportions at all.
The oracle is InPose given each test subject's true bone lengths. It is the ceiling, and every calibrated method is reported as a gap to it. The file gives no reported MPJPE (mean per-joint position error) for InPose on AMASS; the paper only states that error "remained nearly flat across all body sizes". We therefore reproduce both naive and oracle numbers ourselves on identical splits.
For context against fixed-skeleton trackers, AvatarPoser reports 4.18 cm MPJPE on AMASS subsets at 60 Hz. QuestSim reports 3.7 cm MPJPE on its in-house real Quest data, which is not available to us. We retrain AvatarPoser on our training split and evaluate it on scaled and re-proportioned test subjects, where InPose reports that end-to-end models such as BoDiffusion and AvatarJLM produce "phantom squats".
The closed-form heuristic baseline scales a template skeleton using standing head height and controller-to-controller arm span from a brief T-pose. It is cheap and decent, so it is the bar for the parameters it actually measures.
The strongest competitor is an amortized regressor: a transformer trained on AMASS to map the first $T$ seconds of three-point data directly to bone lengths, with its output fed to InPose. It costs one forward pass. It is trained on the same training subjects, receives the same seconds of data including the same prompted motions, and gets matched tuning budget.
Within BO, we compare Sobol-initialized and random-initialized GP-BO at the same evaluation budget, same surrogate, and same acquisition after initialization. We also run gradient descent on $\theta$ through the solver, with gradients averaged over solver samples, at matched wall-clock time.
Three ablations isolate our ingredients. The first removes the decaying population prior. The second replaces the HIPE initialization with Sobol while keeping the prior. The third keeps both but replaces information-ranked prompts with a fixed script of the same total duration.
Beating the competition means a statistically reliable reduction in the oracle gap, averaged over held-out subjects, at a fixed budget of evaluations and seconds. The reduction must be largest on lower-body MPJPE and on out-of-distribution proportions.
Why not just…?
Why not just measure height and arm span in a T-pose? That works for what it measures, and we use it as the prior mean rather than competing with it. It cannot split standing height between torso and legs, or split arm length between upper arm and forearm. The leg split is exactly the lower-body direction where InPose already has its largest error, because three points give no direct position guidance there. The experiment reports per-bone error so readers can see whether the heuristic leaves anything for us to recover. If it does not, the active component is unnecessary.
Why not just train a network to regress bone lengths from the first few seconds? This is the real threat, and it may match us on typical proportions. InPose's evidence is that learned models overfit to the body-scale distribution they were trained on and fail off it. A regressor is also blind to which of its outputs are unconstrained by the data. We settle this with a held-out test on unusual proportions, including scales of 0.6x–1.4x and the long-arms-short-torso cases InPose uses. If the regressor stays within our error there, we drop the BO loop and keep only the active-prompt component, attached to the regressor's own predicted uncertainty.
Why not just use Sobol initialization or gradient descent? HIPE's CTSlice result suggests space-filling designs waste the first batch, but that evidence comes from ML tuning, not kinematic calibration. Gradients through a stochastic diffusion solver are noisy and cost several solver calls each. Both are run at matched budget, so this question is answered empirically rather than assumed.
Proposed Experiments
The main testbed is AMASS. It provides 346 subjects with fitted body-shape parameters, from which ground-truth bone lengths follow. We hold out subjects, synthesize headset and controller streams as in AvatarPoser, and add re-proportioned versions of the held-out subjects across the scale range InPose evaluates on the Transitions and HumanEva data. For a real-sensor check, we evaluate on Ego-Exo4D head and hand trajectories, where DSPoser reports 16.84 cm MPJPE. There we only test whether calibration reduces error relative to the mean skeleton, since its ground-truth skeletons are less precise.
Metrics are per-bone length error in millimeters, full-body and lower-body MPJPE, gap to oracle, number of solver evaluations, seconds of user motion including prompts, and wall-clock calibration time.
The decisive checks are three curves. The first plots oracle gap against solver evaluations for HIPE-plus-prior versus Sobol. The second plots oracle gap against seconds of user motion for information-ranked prompts versus a fixed script. The third compares lower-body MPJPE on out-of-distribution proportions against the amortized regressor. The proposal stands only if the first two curves show a clear early-budget advantage. If the third shows no advantage over the regressor, we report that honestly and narrow the contribution to the active-prompt mechanism.
References
- Zero-shot Human Pose Estimation using Diffusion-based Inverse solvers
- Informed Initialization for Bayesian Optimization and Active Learning
- Incorporating Expert Priors into Bayesian Optimization via Dynamic Mean Decay
- Incorporating Expert Prior in Bayesian Optimisation via Space Warping
- Transformer IMU Calibrator: Dynamic On-body IMU Calibration for Inertial Motion Capture
- Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors
- AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing
- QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars