CenterHMR: Multi-Person Center-based Human Mesh Recovery

Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J. Black, Tao Mei

Abstract

This paper focuses on multi-person 3D mesh recovery from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline that detects bounding boxes and then regresses the body from bounding-box-level features. However, multi-person occlusion and truncation can make these features ambiguous, resulting in recovery failure. To address this, we present a novel bottom-up single-shot method named Center-based Human Mesh Recovery network (CenterHMR), also known as ROMP. The key idea is an explicit center-based representation for bottom-up pixel-level estimation. Guided by body centers, our model effectively locates every person and learns robust features under occlusion. Furthermore, a collision-aware center representation ensures distinguishable distances between body centers in crowded scenes. CenterHMR achieves state-of-the-art performance on four challenging benchmarks and is the first open-source real-time (over 30 FPS) implementation of monocular multi-person 3D mesh recovery.

Introduction: The Challenge of Multi-Person Occlusion

Most 3D human mesh recovery (HMR) methods follow a "top-down" approach: first, detect people using bounding boxes, then crop those boxes and regress the 3D mesh for each person individually. While effective for single-person images, this strategy often fails in the real world. In crowded scenes, bounding boxes overlap significantly, making the features inside a box ambiguous—it becomes unclear which limbs belong to which person. Furthermore, when people are truncated by the image frame, detection often fails entirely.

Traditional detection-based methods struggle with ambiguity when bounding boxes overlap in crowded scenes.

The authors argue that a "bottom-up" approach—one that looks at the whole scene holistically rather than through isolated crops—is necessary for robust, multi-person 3D estimation.

Methodology: The ROMP Framework

The proposed framework, ROMP (Regress all meshes in a One-stage fashion for Multiple 3D People), predicts three dense, differentiable maps from the entire image:

  1. Body Center Heatmap ($C_m$): Predicts the probability of a pixel being the center of a human body (specifically, the average of visible torso joints).
  2. Camera Map ($A_m$): Predicts camera parameters (scale and translation) for a person centered at that pixel.
  3. SMPL Map ($S_m$): Predicts the 142 parameters required for the SMPL body model (pose and shape) for a person at that pixel.
The ROMP pipeline uses explicit pixel-level representations to sample mesh parameters from body center heatmaps.

Collision-Aware Representation (CAR)

A major challenge for bottom-up methods is "center collision"-when two people overlap so much that their centers occupy the same pixel. To solve this, the authors treat body centers like like-charged particles. During training, if two centers are too close, a repulsion field pushes them apart. This forces the model to learn to "sample" mesh parameters from visible body parts rather than a single ambiguous point, significantly improving robustness in crowds.

Experiments and Results

The model was trained on a massive suite of data, including 3D datasets (Human3.6M, MPI-INF-3DHP), pseudo-3D data (UP), and 2D pose datasets (MS COCO, MPII, LSP, AICH). Evaluation focused on challenging benchmarks like 3DPW (in-the-wild), CMU Panoptic (multi-person), and Crowdpose.

  • State-of-the-Art Accuracy: On the 3DPW dataset, ROMP achieved an MPJPE (Mean Per Joint Position Error) of 80.1 mm, significantly outperforming previous multi-stage SOTA methods like VIBE (93.5 mm).
  • Crowd Robustness: On the CMU Panoptic benchmark, ROMP reduced the error by over 10% compared to the previous best multi-person method (CRMH).
  • Real-Time Speed: Because it is a one-stage, anchor-free model, it avoids the computational overhead of region proposals. ROMP runs at roughly 30.9 FPS on a standard NVIDIA 1070Ti GPU, with processing time remaining nearly constant regardless of the number of people in the scene.

Key Insights and Future Work

By looking at the whole image, the model learns spatial relationships between people, allowing it to disambiguate which limbs belong to whom. The CAR module provides a clever way to handle the inherent ambiguity of 2D projections by forcing the model to distinguish between overlapping centers.

While ROMP sets a new standard for real-time multi-person HMR, limitations remain. Complex depth relationships, such as people hugging or intertwining, still pose a challenge for perfect rendering. Future directions include integrating more fine-grained pixel-level depth estimation directly into the one-stage framework to further improve performance in extremely crowded or complex environments.

Summary updated

Abstract

This paper focuses on multi-person 3D mesh recovery from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline, that detects bounding boxes and then regresses the body from bounding-box-level features. However, multi-person occlusion and truncation can make these features ambiguous, which results in the failure of recovery. To deal with this problem, we present a novel bottom-up single-shot method, named Center-based Human Mesh Recovery network (CenterHMR). The key idea is to develop an explicit center-based representation for bottom-up pixel-level estimation. Guided by the body centers, our model effectively locates every person and learns robust and discriminative features under occlusion. In an end-to-end manner, the model is trained to estimate multiple differentiable maps that contain the information of multi-person 3D body meshes and their locations.Furthermore, when encountering severe multi-person occlusion, the body centers may be very close or even overlapping. A collision-aware center representation is developed to ensure a distinguishable distance between body centers. Our proposed CenterHMR achieves state-of-the-art performance on four challenging multi-person/occlusion benchmarks (3DPW, CMU Panoptic, MuPoTs-3D, and 3DOH50K). Experiments on crowded/occluded datasets demonstrate the stability under various types of occlusion. Due to the concise bottom-up single-shot design, our released demo code (this https URL) is the first open-source real-time (over 30 FPS) implementation of monocular multi-person 3D mesh recovery.
BibTeX
@inproceedings{sun2020centerhmr,
  author = {Yu Sun and Qian Bao and Wu Liu and Yili Fu and Michael J. Black and Tao Mei},
  title = {{CenterHMR: Multi-Person Center-based Human Mesh Recovery}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2020},
  doi = {10.48550/arxiv.2008.12272},
}
Figure