Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Grégory Rogez, Thomas Lucas, Philippe Weinzaepfel

Introduction

Human Mesh Recovery (HMR) involves reconstructing a 3D model of a person from a 2D image. Traditionally, this has been a "top-down" multi-stage process: first, detect people; second, crop the image around them; and third, run a specialized model on each crop to estimate pose and shape.

While accurate, this approach suffers from significant efficiency bottlenecks, context loss due to cropping, and increased complexity when attempting "whole-body" recovery (including hands and faces). Multi-HMR addresses these challenges by proposing a single-shot architecture that detects and reconstructs everyone in the scene simultaneously, including fine-grained details like finger positions and facial expressions.

Table 1: Comparison of Multi-HMR against existing single-person and multi-person methods. Multi-HMR is the first to achieve whole-body, single-shot, camera-aware recovery.

Methodology

The architecture of Multi-HMR is built on three main pillars designed to maintain global context while capturing local detail:

  • ViT Backbone: The model utilizes a standard Vision Transformer (initialized with DINOv2 weights) to extract features. Unlike previous convolutional models, the ViT maintains a global field of view across all patches, which is critical for understanding the spatial relationships between multiple people.
  • Human Prediction Head (HPH): This novel component is the core of the single-shot capability. After detecting people via a coarse 2D heatmap, the model uses a cross-attention mechanism. Each detected person is represented by a "query" token that attends to all image features. This allows the model to leverage both local details (for hands) and global context (for 3D placement) without requiring high-resolution crops.
  • Camera Awareness: To handle varying camera lenses, the authors optionally encode "camera ray directions" as Fourier features and add them to the image tokens. This helps the model understand perspective distortion, significantly improving 3D distance estimation.
The Multi-HMR pipeline: An input image is processed by a ViT backbone, followed by patch-level detection and a Human Prediction Head that outputs SMPL-X parameters for all individuals.

Datasets and Training

The authors utilize a mix of synthetic and real-world data to bridge the sim-to-real gap. Training includes large-scale datasets like BEDLAM, AGORA, and UBody.

Crucially, the authors introduce CUFFS (Close-Up Frames of Full-body Subjects). Because single-shot models often struggle with hands—which appear very small in full-body images—CUFFS provides 60,000 synthetic images of humans close to the camera with diverse, clearly visible hand poses. This "booster" dataset was key to achieving high-quality hand reconstruction without using specialized crops.

Key Results and Insights

Multi-HMR demonstrates that high-resolution crops are not strictly necessary for detailed reconstruction. If a model can attend to the entire feature map for each person via cross-attention, it can "find" the necessary details for hands and faces even in a single-shot pass.

  • State-of-the-Art Performance: Multi-HMR outperforms existing multi-person and whole-body methods on major benchmarks like EHF (whole-body) and AGORA.
  • Efficiency: A version using a ViT-Small backbone at 448x448 resolution runs at 30 FPS on an NVIDIA V100 GPU, making it viable for real-time applications.
  • 3D Accuracy: By incorporating camera intrinsics, the model reduced the Mean Root Position Error (MRPE)—a measure of how accurately people are placed in 3D space—by over 50% on some datasets compared to camera-agnostic versions.

Limitations and Future Work

Despite its strengths, the model has limitations. It detects people at a patch level; if two people's centers fall into the same 14x14 pixel patch, the model may only detect one. Additionally, the model currently relies on seeing the head for detection; if the head is out of frame, detection fails. Future research may explore alternative pose representations to reduce error accumulation down the kinematic chain, such as in the transition from shoulder to hand.

Summary updated

Abstract

We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e., including hands and facial expressions, using the SMPL-X parametric model and 3D location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person locations, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and 3D location using a new cross-attention module called the Human Prediction Head (HPH), with one query attending to the entire set of features for each detected person. As direct prediction of fine-grained hands and facial poses in a single shot, i.e., without relying on explicit crops around body parts, is hard to learn from existing data, we introduce CUFFS, the Close-Up Frames of Full-Body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating it into the training data further enhances predictions, particularly for hands. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously: a ViT-S backbone on 448×448 images already yields a fast and competitive model, while larger models and higher resolutions obtain state-of-the-art results.
BibTeX
@inproceedings{baradel2024multi,
  author = {Fabien Baradel and Matthieu Armando and Salma Galaaoui and Romain Brégier and Grégory Rogez and Thomas Lucas and Philippe Weinzaepfel},
  title = {{Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73337-6\_12},
}
Figure