Introduction
Human Mesh Recovery (HMR) involves reconstructing a 3D model of a person from a 2D image. Traditionally, this has been a "top-down" multi-stage process: first, detect people; second, crop the image around them; and third, run a specialized model on each crop to estimate pose and shape.
While accurate, this approach suffers from significant efficiency bottlenecks, context loss due to cropping, and increased complexity when attempting "whole-body" recovery (including hands and faces). Multi-HMR addresses these challenges by proposing a single-shot architecture that detects and reconstructs everyone in the scene simultaneously, including fine-grained details like finger positions and facial expressions.
Methodology
The architecture of Multi-HMR is built on three main pillars designed to maintain global context while capturing local detail:
- ViT Backbone: The model utilizes a standard Vision Transformer (initialized with DINOv2 weights) to extract features. Unlike previous convolutional models, the ViT maintains a global field of view across all patches, which is critical for understanding the spatial relationships between multiple people.
- Human Prediction Head (HPH): This novel component is the core of the single-shot capability. After detecting people via a coarse 2D heatmap, the model uses a cross-attention mechanism. Each detected person is represented by a "query" token that attends to all image features. This allows the model to leverage both local details (for hands) and global context (for 3D placement) without requiring high-resolution crops.
- Camera Awareness: To handle varying camera lenses, the authors optionally encode "camera ray directions" as Fourier features and add them to the image tokens. This helps the model understand perspective distortion, significantly improving 3D distance estimation.
Datasets and Training
The authors utilize a mix of synthetic and real-world data to bridge the sim-to-real gap. Training includes large-scale datasets like BEDLAM, AGORA, and UBody.
Crucially, the authors introduce CUFFS (Close-Up Frames of Full-body Subjects). Because single-shot models often struggle with hands—which appear very small in full-body images—CUFFS provides 60,000 synthetic images of humans close to the camera with diverse, clearly visible hand poses. This "booster" dataset was key to achieving high-quality hand reconstruction without using specialized crops.
Key Results and Insights
Multi-HMR demonstrates that high-resolution crops are not strictly necessary for detailed reconstruction. If a model can attend to the entire feature map for each person via cross-attention, it can "find" the necessary details for hands and faces even in a single-shot pass.
- State-of-the-Art Performance: Multi-HMR outperforms existing multi-person and whole-body methods on major benchmarks like EHF (whole-body) and AGORA.
- Efficiency: A version using a ViT-Small backbone at 448x448 resolution runs at 30 FPS on an NVIDIA V100 GPU, making it viable for real-time applications.
- 3D Accuracy: By incorporating camera intrinsics, the model reduced the Mean Root Position Error (MRPE)—a measure of how accurately people are placed in 3D space—by over 50% on some datasets compared to camera-agnostic versions.
Limitations and Future Work
Despite its strengths, the model has limitations. It detects people at a patch level; if two people's centers fall into the same 14x14 pixel patch, the model may only detect one. Additionally, the model currently relies on seeing the head for detection; if the head is out of frame, detection fails. Future research may explore alternative pose representations to reduce error accumulation down the kinematic chain, such as in the transition from shoulder to hand.
Summary updated