TL;DR
MetricPose is a unified 3D human pose estimator that resolves the inherent size-distance ambiguity of monocular images by flexibly integrating RGB, depth, and ground geometry cues. By extending a foundation model with optional metric-processing modules, it achieves metrically accurate pose and placement in camera coordinates, reducing camera-local joint error by up to 54% compared to state-of-the-art baselines.
What Problem Does This Paper Address?
Most 3D human pose estimation (HPE) research focuses on "root-relative" pose—essentially, the shape of the body after centering it at the pelvis. While useful, this ignores the absolute scale and distance of the person from the camera. In monocular RGB images, a "size-distance ambiguity" exists: a small child close to the camera can project the same image as an adult standing further away.
For applications like robotics (where a robot needs to know exactly how far away a person is to avoid a collision), biomechanics, or social interaction analysis, this metric accuracy is critical. Existing solutions are often rigid: they either rely solely on learned RGB priors (which fail on unusual body sizes like children) or require depth data at all times, making them unusable in RGB-only scenarios.
Methods
The authors introduce MetricPose, which builds upon the SAM 3D Body (3DB) foundation model. The key innovation is a unified architecture that uses a single set of weights to handle three input modes: RGB-only, RGB + Ground geometry, and RGB + Depth.
Key technical components include:
- LoRA Adaptation: They use Low-Rank Adaptation (LoRA) to fine-tune the pretrained image encoder and body decoder, keeping the bulk of the foundation model's knowledge intact while specializing it for metric tasks.
- Gated Depth Fusion: Depth information is encoded and fused into the RGB feature tokens using a channel-wise gating mechanism. This allows the model to "ignore" the depth branch if data is missing.
- Two-Pass Refinement: A "Depth-at-joint refiner" compares predicted joint depths with actual measured surface depths to iteratively correct the pose.
- Deterministic Geometric Corrections: Instead of just learning from data, the model uses hard geometric constraints. For example, if the ground plane is known, it rescales the entire body and its distance to ensure the feet touch the floor without changing the 2D image projection.
Datasets
The model was fine-tuned on BEDLAM2 and CMU Panoptic, which provide high-quality registered RGB and depth data.
Evaluation was conducted across a wide range of benchmarks to test different capabilities:
- Standard benchmarks: 3DPW, EMDB, and RICH.
- Depth-enabled benchmarks: HuMMan, EgoBody, and held-out sets of BEDLAM2 and CMU Panoptic.
- Specialized scenarios: CMU Panoptic Toddler sequences (to test scale ambiguity) and OpenCap (a smartphone-based clinical motion capture dataset).
- Simulation: A controlled experiment in Isaac Sim using a 1.12m-tall child avatar to precisely measure distance errors.
Key Insights and Ideas
- Complementary Cues: The authors show that RGB priors, depth, and ground geometry are complementary. RGB priors provide the basic shape, depth fixes local joint placement and absolute distance, and ground geometry provides a powerful global constraint on scale.
- Preserving the 2D Projection: A major "aha" moment is the use of deterministic rescaling. By jointly adjusting body scale and camera translation, the model can fix metric errors while perfectly maintaining the 2D alignment of the pose in the image.
- Modality Dropout: By training with "modality dropout" (randomly removing depth during training), they created a single model that doesn't lose its RGB-only performance when depth is unavailable, yet significantly improves when it is.
- The "Camera-Local" Metric: The paper argues that standard metrics like PA-MPJPE (which uses Procrustes alignment) hide massive scale errors. They champion "camera-local" MPJPE as the essential metric for real-world applications.
Key Results
MetricPose significantly outperforms the SAM 3D Body baseline across all modes:
- Error Reduction: Compared to 3DB, MetricPose reduces camera-local joint-position error by 12.16% (RGB), 31.00% (RGB-Ground), and 54.61% (RGB-D).
- Solving Scale Ambiguity: In the Isaac Sim child-avatar test, 3DB misestimated distance by over 4 meters at a 5-meter range. MetricPose's RGB-D mode reduced this error to just 7 cm.
- Toddler Accuracy: On toddler sequences, MetricPose (RGB-D) reduced camera-local error from 2001 mm (3DB) to just 97 mm, outperforming specialized all-age models like Anny-Fit by 91%.
Limitations and Future Work
- Sensitivity: The model remains sensitive to calibration errors and misregistered depth data.
- Assumptions: The ground correction module assumes a planar floor and requires known camera height and gravity direction, which may not always be available in "in-the-wild" monocular videos.
- Computation: Adding the depth-processing branch increases the computational overhead compared to the base RGB model.
- Future Directions: The authors plan to explore uncertainty-aware fusion (letting the model decide how much to trust the depth) and integrating temporal or multi-view cues for even greater stability.
Summary updated