Abstract
Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust (MeTRo) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D.
Introduction
Monocular 3D human pose estimation—predicting anatomical landmarks in millimeters from a single image—is a fundamental challenge in computer vision. Traditional approaches often struggle with two primary geometric ambiguities: scale ambiguity and truncation. Scale ambiguity arises because a small person close to the camera can appear identical to a large person further away. Truncation occurs when body parts are outside the image frame; standard pixel-aligned heatmaps cannot predict joints they cannot "see."
MeTRAbs addresses these issues by introducing a novel 3D heatmap representation (MeTRo) that is defined in metric space (meters) rather than image space (pixels). This allows models to directly estimate scale-correct 3D poses and "hallucinate" body joints that are truncated or outside the image frame.
The MeTRo Representation
The core innovation is the MeTRo (Metric-Scale Truncation-Robust) volumetric heatmap. In traditional 2.5D heatmaps, the X and Y axes correspond to image pixels. In contrast, MeTRo heatmaps are defined within a fixed 3D metric volume (e.g., a 2.2m cube) centered around the subject. By interpreting heatmap axes as meters, the network learns to implicitly reason about scale and perspective back-projection within its weights.
This decoupling from image boundaries allows the model to perform "hallucination through context." Because CNNs use zero-padding, they can perceive image borders, allowing the MeTRo representation to infer the position of joints outside the frame based on visible body parts (e.g., estimating an ankle's position from a visible knee).
MeTRAbs Architecture
The MeTRAbs architecture utilizes a "two-headed" CNN, typically built upon a ResNet-50 backbone. One head predicts the 3D MeTRo heatmaps for root-relative metric pose, while the second head predicts standard 2D image-space heatmaps.
These predictions are combined through a Differentiable Absolute Pose Recovery module. This module uses a differentiable linear least-squares solver based on the pinhole camera model, allowing the entire network to be supervised directly on final absolute 3D coordinates in the camera frame. The authors found that this absolute pose loss is crucial for accurate distance estimation.
Experimental Results
The model was evaluated on major benchmarks including Human3.6M, MPI-INF-3DHP, and MuPoTS-3D. MeTRAbs achieved state-of-the-art performance, notably reaching 49.3 mm MPJPE on Human3.6M using a simple ResNet-50 backbone.
A key strength of the system is its robustness to extreme truncation. The model maintains stable performance even when multiple joints are truncated, accurately completing skeletons when only the upper body is visible. Furthermore, the root-relative version is highly efficient, capable of processing over 500 crops per second on modern hardware.
Conclusion
MeTRAbs demonstrates that heatmaps do not need to be pixel-aligned to be effective. By shifting to a metric-scale representation, the model overcomes traditional limitations regarding scale and truncation. While absolute distance estimation still requires knowledge of camera focal lengths, the MeTRo representation provides a powerful framework for robust, real-time 3D human pose estimation in the wild.
Summary updated