Abstract
This paper focuses on the regression of multiple 3D people from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline that first detects people in bounding boxes and then independently regresses their 3D body meshes. In contrast, we propose to Regress all meshes in a One-stage fashion for Multiple 3D People (termed ROMP). The approach is conceptually simple, bounding box-free, and able to learn a per-pixel representation in an end-to-end manner. Our method simultaneously predicts a Body Center heatmap and a Mesh Parameter map, which can jointly describe the 3D body mesh on the pixel level. Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map. Equipped with such a fine-grained representation, our one-stage framework is free of the complex multi-stage process and more robust to occlusion. Compared with state-of-the-art methods, ROMP achieves superior performance on the challenging multi-person benchmarks, including 3DPW and CMU Panoptic. Experiments on crowded/occluded datasets demonstrate the robustness under various types of occlusion. The released code is the first real-time implementation of monocular multi-person 3D mesh regression.
Introduction: The Limitations of Top-Down Approaches
Most existing 3D human pose and shape estimation (HPS) methods follow a "top-down," multi-stage pipeline. This process typically begins with a 2D detector finding people and drawing bounding boxes, followed by a second model that crops these boxes to regress individual 3D meshes.
However, this approach suffers from three major flaws. First, it is computationally expensive due to repeated feature extraction for every detected box. Second, it faces significant ambiguity in crowds; when people overlap, their bounding boxes become nearly identical, making it difficult for the model to distinguish which features belong to which individual. Finally, truncation at the edge of a frame often leads to failure because the model lacks the local context required for accurate regression.
Methodology: One-Stage Regression (ROMP)
To overcome these issues, we propose ROMP, a "bottom-up," one-stage framework that treats 3D mesh recovery as a dense pixel-level prediction task. Unlike multi-stage methods that only see a cropped box, ROMP's one-stage approach maintains a holistic view of the entire scene, allowing it to use global context to reason about occluded or truncated body parts.
The Architecture
Using a standard backbone (such as ResNet-50 or HRNet), the model predicts two primary maps:
- Body Center Heatmap ($C_m$): Predicts the probability that a specific pixel is the center of a human body.
- Mesh Parameter Map ($P_m$): A dense map where each pixel contains a 145-dimensional vector. This vector includes SMPL parameters (pose and shape) and camera parameters (scale and translation).
During inference, the system identifies local maxima in the heatmap and "samples" the 3D parameters from the corresponding coordinates in the parameter map. By moving to this center-based, anchor-free representation, we eliminate the need for thousands of pre-defined anchors or complex post-processing grouping steps.
Handling Crowds with Collision-Aware Representation
A fundamental challenge in bottom-up methods is the "center collision" problem, where two people overlap so much that their centers occupy the same pixel. To solve this, we introduced a Collision-Aware Representation (CAR) using a repulsion field. During training, centers of overlapping people are treated like like-charged particles that push each other apart. This forces the model to represent people using pixels located on their visible body parts, significantly reducing feature ambiguity in crowded environments.
Key Results and Performance
ROMP achieves state-of-the-art accuracy and real-time performance, running at over 30 FPS on a single NVIDIA 1070Ti GPU. Crucially, its processing time remains roughly constant regardless of the number of people in the image.
- Superior Accuracy: ROMP outperformed previous state-of-the-art methods (like VIBE and SPIN) on the challenging 3DPW and CMU Panoptic benchmarks.
- Robustness to Occlusion: On the Crowdpose dataset, the inclusion of the CAR module improved the Average Precision (AP) from 33.9 to 59.7, a massive jump for crowded scenes.
Conclusion and Future Work
ROMP represents a breakthrough in efficient, multi-person 3D mesh recovery. While it handles 2D overlaps brilliantly, extremely complex depth ordering (e.g., human pyramids) remains a challenge. Future work will explore more explicit pixel-level depth estimation, the integration of temporal information for video smoothing, and interaction modeling for cases where people are in physical contact.
Summary updated