TL;DR
FAB (Fast and Accurate Body estimation) is a real-time system that reconstructs 3D human pose and shape by prioritizing the prediction of dense surface points over abstract model parameters. By distilling a massive foundation model into a more efficient architecture, FAB achieves state-of-the-art accuracy in camera, world, and scene spaces while running at ~40 FPS on consumer-grade GPUs.
What Problem Does This Paper Address?
Most 3D human pose and shape estimation (HPS) methods rely on regressing parameters for a model like SMPL. While convenient, this often leads to a "prior-vs-evidence" conflict: the fitted mesh might look plausible but fails to align perfectly with the 2D pixels (e.g., feet sliding or joints not matching the image). Furthermore, high-accuracy models are often too slow for real-time applications, while fast models sacrifice the physical "grounding" needed for robotics and AR.
FAB tackles this by asking: can we build a system that is fast enough for real-time use, accurate enough to beat heavy foundation models, and robust enough to drive a humanoid robot in a complex 3D scene?
Methods
The core philosophy of FAB is "points-first" estimation. Instead of jumping straight to mesh parameters, the network predicts dense 2D and 3D body points (keypoints, skeleton joints, and surface points) directly from the image.
- Architecture & Distillation: The authors start with the powerful but slow SAM 3D Body model. They distill its knowledge into a more efficient student model, FAB-L (using a ViT-L backbone).
- Point Prediction & Fitting: The model predicts 2D/3D heatmaps for 70 keypoints and 468 surface points. It then fits the SMPL mesh to these points in a single learned pass followed by two fast Gauss-Newton optimization steps. This ensures the final mesh is anchored to the actual visual evidence.
- Contacts as First-Class Citizens: FAB explicitly predicts contacts for heels, toes, hands, and knees. These aren't just labels; they are used to "glue" the human model to reconstructed 3D scenes, preventing the common "floating" or "sinking" artifacts.
- Human-to-Humanoid Retargeting: The system includes FAB-Retarget, which maps the estimated human motion onto a Unitree G1 humanoid. It uses the dense points to ensure the robot's bone directions and surface alignment closely mirror the human's.
Datasets
The authors evaluate FAB across a wide array of benchmarks to test different capabilities:
- Camera Space: 3DPW, RICH, EgoBody, Harmony4D, and SSP-3D.
- World Space & Scenes: EMDB-2, RICH-191, SLOPER4D, and PROX.
- Robustness: They introduced FAB-Hard, a curated benchmark focusing on low light, extreme acrobatics, and heavy occlusion.
Key Insights and Ideas
- Points over Parameters: Predicting points directly allows the model to be more "visually faithful." If a hand is at a certain pixel, the point prediction puts it there, whereas a parameter-only model might let the mesh drift due to internal constraints.
- The Power of Distillation: FAB-L is 4.6× faster than its teacher (SAM 3D Body) in eager PyTorch but actually more accurate on most benchmarks. This suggests that with the right training recipe, smaller models can effectively internalize the complex priors of massive foundation models.
- Scene Integration via Contacts: By fusing static scene depth and aligning it using predicted foot contacts, FAB creates a metric, gravity-aligned world where the human and the environment actually interact realistically.
Key Results
- Speed: FAB-L runs at 39.2 FPS on an NVIDIA RTX 4070 Ti SUPER, covering the entire pipeline from detection to mesh fitting. A smaller version, FAB-S, reaches 63.8 FPS.
- Accuracy: On the RICH benchmark, FAB-L achieves a PA-MPJPE (Procrustes-Aligned Mean Per Joint Position Error) of 27.8 mm, outperforming much heavier models like SAM 3D Body (31.3 mm).
- Robustness: In the FAB-Hard "extreme acrobatics" category, FAB-L maintains a 2D accuracy ([email protected]) of 79.6%, significantly higher than the baseline SAM 3D Body's 76.2%.
- Retargeting: FAB-Retarget's references follow human bone directions more closely than specialized state-of-the-art retargeters like SOMA or GMR.
Limitations and Future Work
- Backbone Scaling: While FAB-L is fast, the authors note that training even larger backbones (like ViT-H) end-to-end was limited by available compute.
- Challenging Poses: The model still struggles with highly unusual poses found in yoga or gymnastics, likely due to a lack of such data in the training mixture.
- Real-Robot Gap: While the retargeting works impressively in simulation (MuJoCo), the authors have not yet tested the system on a physical robot.
- World-Space Drift: In very long sequences with significant camera movement, the world-space estimation can still suffer from drift.
For more details, visit the project page.
Summary updated