Fast and Accurate Body Estimation in Pixels, World Space and Scenes

Jinkun CaoarXiv 2026 · arXiv

TL;DR

FAB (Fast and Accurate Body estimation) is a real-time system that reconstructs 3D human pose and shape by prioritizing the prediction of dense surface points over abstract model parameters. By distilling a massive foundation model into a more efficient architecture, FAB achieves state-of-the-art accuracy in camera, world, and scene spaces while running at ~40 FPS on consumer-grade GPUs.

What Problem Does This Paper Address?

Most 3D human pose and shape estimation (HPS) methods rely on regressing parameters for a model like SMPL. While convenient, this often leads to a "prior-vs-evidence" conflict: the fitted mesh might look plausible but fails to align perfectly with the 2D pixels (e.g., feet sliding or joints not matching the image). Furthermore, high-accuracy models are often too slow for real-time applications, while fast models sacrifice the physical "grounding" needed for robotics and AR.

FAB tackles this by asking: can we build a system that is fast enough for real-time use, accurate enough to beat heavy foundation models, and robust enough to drive a humanoid robot in a complex 3D scene?

Methods

The core philosophy of FAB is "points-first" estimation. Instead of jumping straight to mesh parameters, the network predicts dense 2D and 3D body points (keypoints, skeleton joints, and surface points) directly from the image.

  1. Architecture & Distillation: The authors start with the powerful but slow SAM 3D Body model. They distill its knowledge into a more efficient student model, FAB-L (using a ViT-L backbone).
  2. Point Prediction & Fitting: The model predicts 2D/3D heatmaps for 70 keypoints and 468 surface points. It then fits the SMPL mesh to these points in a single learned pass followed by two fast Gauss-Newton optimization steps. This ensures the final mesh is anchored to the actual visual evidence.
  3. Contacts as First-Class Citizens: FAB explicitly predicts contacts for heels, toes, hands, and knees. These aren't just labels; they are used to "glue" the human model to reconstructed 3D scenes, preventing the common "floating" or "sinking" artifacts.
  4. Human-to-Humanoid Retargeting: The system includes FAB-Retarget, which maps the estimated human motion onto a Unitree G1 humanoid. It uses the dense points to ensure the robot's bone directions and surface alignment closely mirror the human's.
The FAB pipeline: from input video to 3D points, fitted mesh, scene alignment, and humanoid retargeting.

Datasets

The authors evaluate FAB across a wide array of benchmarks to test different capabilities:

  • Camera Space: 3DPW, RICH, EgoBody, Harmony4D, and SSP-3D.
  • World Space & Scenes: EMDB-2, RICH-191, SLOPER4D, and PROX.
  • Robustness: They introduced FAB-Hard, a curated benchmark focusing on low light, extreme acrobatics, and heavy occlusion.

Key Insights and Ideas

  • Points over Parameters: Predicting points directly allows the model to be more "visually faithful." If a hand is at a certain pixel, the point prediction puts it there, whereas a parameter-only model might let the mesh drift due to internal constraints.
  • The Power of Distillation: FAB-L is 4.6× faster than its teacher (SAM 3D Body) in eager PyTorch but actually more accurate on most benchmarks. This suggests that with the right training recipe, smaller models can effectively internalize the complex priors of massive foundation models.
  • Scene Integration via Contacts: By fusing static scene depth and aligning it using predicted foot contacts, FAB creates a metric, gravity-aligned world where the human and the environment actually interact realistically.
FAB's output in a world-space scene, showing the reconstructed environment and the retargeted G1 humanoid.

Key Results

  • Speed: FAB-L runs at 39.2 FPS on an NVIDIA RTX 4070 Ti SUPER, covering the entire pipeline from detection to mesh fitting. A smaller version, FAB-S, reaches 63.8 FPS.
  • Accuracy: On the RICH benchmark, FAB-L achieves a PA-MPJPE (Procrustes-Aligned Mean Per Joint Position Error) of 27.8 mm, outperforming much heavier models like SAM 3D Body (31.3 mm).
  • Robustness: In the FAB-Hard "extreme acrobatics" category, FAB-L maintains a 2D accuracy ([email protected]) of 79.6%, significantly higher than the baseline SAM 3D Body's 76.2%.
  • Retargeting: FAB-Retarget's references follow human bone directions more closely than specialized state-of-the-art retargeters like SOMA or GMR.
Comparison of speed vs. accuracy across various state-of-the-art models.

Limitations and Future Work

  • Backbone Scaling: While FAB-L is fast, the authors note that training even larger backbones (like ViT-H) end-to-end was limited by available compute.
  • Challenging Poses: The model still struggles with highly unusual poses found in yoga or gymnastics, likely due to a lack of such data in the training mixture.
  • Real-Robot Gap: While the retargeting works impressively in simulation (MuJoCo), the authors have not yet tested the system on a physical robot.
  • World-Space Drift: In very long sequences with significant camera movement, the world-space estimation can still suffer from drift.

For more details, visit the project page.

Summary updated

Abstract

We present FAB (Fast and Accurate Body estimation), a human body estimator that jointly improves inference efficiency and body-recovery fidelity. Rather than predicting only parametric mesh-model parameters, FAB directly predicts human skeleton joints and dense surface points. In images, it predicts dense 2D and 3D body points alongside a parametric Momentum Human Rig (MHR) head, then fits SMPL in one learned pass followed by two Gauss-Newton steps. Its ViT-L student, FAB-L, distilled from a fine-tuned SAM 3D Body, runs in real time for one person on an RTX 4070 Ti SUPER, from detection to a fitted body, at 4.6x the speed of SAM 3D Body's eager-PyTorch body-only path, while improving accuracy on most camera-space benchmarks. The same network predicts heel, toe, hand, and knee contacts, which connect the body to the scene. FAB also estimates world-space motion, fuses static-scene depth around the visible person, and aligns body and scene using predicted contacts. For human-to-humanoid real-to-sim transfer, an in-model G1 head and FAB-Retarget, a motion-level retargeter that reads dense points or the fitted mesh, map motion to a Unitree G1. FAB-Retarget produces references that follow the person more closely than specialized retargeters including SOMA, GMR, UMR, and OmniRetarget, and substantially more closely than Fast SAM 3D Body retargeted with our sequence IK. Project page: https://www.jinkuncao.com/FAB.
BibTeX
@misc{cao2026fast,
  author = {Jinkun Cao},
  title = {{Fast and Accurate Body Estimation in Pixels, World Space and Scenes}},
  year = {2026},
  eprint = {2611.25801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25801},
}
Figure