MetricPose: Flexible 3D Human Pose Estimation with Depth and Geometric Cues

Shifan Zhu, Sabad Modi, Hang Yang, Arushi Agrawal, Steven Gao, Meghan Huber, Nathan Wycoff, Donghyun KimarXiv 2026 · arXiv

TL;DR

MetricPose is a unified 3D human pose estimator that resolves the inherent size-distance ambiguity of monocular images by flexibly integrating RGB, depth, and ground geometry cues. By extending a foundation model with optional metric-processing modules, it achieves metrically accurate pose and placement in camera coordinates, reducing camera-local joint error by up to 54% compared to state-of-the-art baselines.

What Problem Does This Paper Address?

Most 3D human pose estimation (HPE) research focuses on "root-relative" pose—essentially, the shape of the body after centering it at the pelvis. While useful, this ignores the absolute scale and distance of the person from the camera. In monocular RGB images, a "size-distance ambiguity" exists: a small child close to the camera can project the same image as an adult standing further away.

For applications like robotics (where a robot needs to know exactly how far away a person is to avoid a collision), biomechanics, or social interaction analysis, this metric accuracy is critical. Existing solutions are often rigid: they either rely solely on learned RGB priors (which fail on unusual body sizes like children) or require depth data at all times, making them unusable in RGB-only scenarios.

Methods

The authors introduce MetricPose, which builds upon the SAM 3D Body (3DB) foundation model. The key innovation is a unified architecture that uses a single set of weights to handle three input modes: RGB-only, RGB + Ground geometry, and RGB + Depth.

Fig. 2. Unified architecture for RGB, RGB-Ground, and RGB-D with shared weights. SAM 3D Body provides the body and camera estimate; optional ground geometry applies a ground-contact correction, while optional depth adds feature fusion, joint-level pose refinement, and depth-surface metric correction.

Key technical components include:

  • LoRA Adaptation: They use Low-Rank Adaptation (LoRA) to fine-tune the pretrained image encoder and body decoder, keeping the bulk of the foundation model's knowledge intact while specializing it for metric tasks.
  • Gated Depth Fusion: Depth information is encoded and fused into the RGB feature tokens using a channel-wise gating mechanism. This allows the model to "ignore" the depth branch if data is missing.
  • Two-Pass Refinement: A "Depth-at-joint refiner" compares predicted joint depths with actual measured surface depths to iteratively correct the pose.
  • Deterministic Geometric Corrections: Instead of just learning from data, the model uses hard geometric constraints. For example, if the ground plane is known, it rescales the entire body and its distance to ensure the feet touch the floor without changing the 2D image projection.

Datasets

The model was fine-tuned on BEDLAM2 and CMU Panoptic, which provide high-quality registered RGB and depth data.

Evaluation was conducted across a wide range of benchmarks to test different capabilities:

  • Standard benchmarks: 3DPW, EMDB, and RICH.
  • Depth-enabled benchmarks: HuMMan, EgoBody, and held-out sets of BEDLAM2 and CMU Panoptic.
  • Specialized scenarios: CMU Panoptic Toddler sequences (to test scale ambiguity) and OpenCap (a smartphone-based clinical motion capture dataset).
  • Simulation: A controlled experiment in Isaac Sim using a 1.12m-tall child avatar to precisely measure distance errors.

Key Insights and Ideas

  • Complementary Cues: The authors show that RGB priors, depth, and ground geometry are complementary. RGB priors provide the basic shape, depth fixes local joint placement and absolute distance, and ground geometry provides a powerful global constraint on scale.
  • Preserving the 2D Projection: A major "aha" moment is the use of deterministic rescaling. By jointly adjusting body scale and camera translation, the model can fix metric errors while perfectly maintaining the 2D alignment of the pose in the image.
  • Modality Dropout: By training with "modality dropout" (randomly removing depth during training), they created a single model that doesn't lose its RGB-only performance when depth is unavailable, yet significantly improves when it is.
  • The "Camera-Local" Metric: The paper argues that standard metrics like PA-MPJPE (which uses Procrustes alignment) hide massive scale errors. They champion "camera-local" MPJPE as the essential metric for real-world applications.

Key Results

MetricPose significantly outperforms the SAM 3D Body baseline across all modes:

  • Error Reduction: Compared to 3DB, MetricPose reduces camera-local joint-position error by 12.16% (RGB), 31.00% (RGB-Ground), and 54.61% (RGB-D).
  • Solving Scale Ambiguity: In the Isaac Sim child-avatar test, 3DB misestimated distance by over 4 meters at a 5-meter range. MetricPose's RGB-D mode reduced this error to just 7 cm.
  • Toddler Accuracy: On toddler sequences, MetricPose (RGB-D) reduced camera-local error from 2001 mm (3DB) to just 97 mm, outperforming specialized all-age models like Anny-Fit by 91%.
Simulation results showing how ground and depth cues drastically reduce distance estimation errors for a child-sized avatar.

Limitations and Future Work

  • Sensitivity: The model remains sensitive to calibration errors and misregistered depth data.
  • Assumptions: The ground correction module assumes a planar floor and requires known camera height and gravity direction, which may not always be available in "in-the-wild" monocular videos.
  • Computation: Adding the depth-processing branch increases the computational overhead compared to the base RGB model.
  • Future Directions: The authors plan to explore uncertainty-aware fusion (letting the model decide how much to trust the depth) and integrating temporal or multi-view cues for even greater stability.

Summary updated

Abstract

In this paper, we present a 3D human pose estimator that predicts metrically accurate human joint positions in the camera coordinate frame. This task requires recovering the configuration of the body joints, scale, and metric camera-local placement. In monocular images, body size and camera distance are coupled through perspective projection and therefore cannot be determined uniquely from image alone. This ambiguity can be reduced by combining three complementary sources of information: learned RGB priors, direct depth measurements, and geometric constraints (e.g., ground contact) from the environment. We extend a pretrained full-body estimator with optional metric depth and geometry-processing modules that improve camera-distance estimation and depth-guided body refinement while preserving RGB-only operation. The resulting unified model supports RGB, RGB-Ground, and RGB-D using shared learned parameters. Unlike pelvis-relative error, camera-local error also measures placement relative to the camera. Compared with the existing foundation model SAM 3D Body, our methods with RGB, RGB-Ground, and RGB-D modes reduce camera-local joint-position error by 12.16%, 31.00% and 54.61% respectively.
BibTeX
@misc{zhu2026metricpose,
  author = {Shifan Zhu and Sabad Modi and Hang Yang and Arushi Agrawal and Steven Gao and Meghan Huber and Nathan Wycoff and Donghyun Kim},
  title = {{MetricPose: Flexible 3D Human Pose Estimation with Depth and Geometric Cues}},
  year = {2026},
  eprint = {2611.08954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08954},
}
Figure