Abstract
This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and physical condition. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance using their canonical atlas. We demonstrate that HUMBI is highly effective in learning and reconstructing a complete human model and is complementary to the existing datasets of human body expressions with limited views and subjects such as MPII-Gaze, Multi-PIE, Human3.6M, and Panoptic Studio datasets.
Introduction: The Challenge of Digital Human Modeling
Creating photorealistic digital humans is a cornerstone for the future of telepresence, AR/VR, and gaming. However, modeling the human form remains a significant challenge due to the complex interplay of geometry, lighting, and viewpoint. To build models that generalize across the global population, researchers require data that covers two critical dimensions: subject diversity (varied ages, ethnicities, and body shapes) and viewpoint density (capturing the same moment from numerous angles).
Existing datasets often compromise on one of these factors. For instance, the MPII-Gaze dataset offers many images but limited subjects, while the Panoptic Studio provides high camera density but lacks subject variety. HUMBI (Human Multiview Behavioral Integration) bridges this gap by providing a "terascale" dataset that excels in both diversity and density.
The HUMBI Dataset: Scaling Diversity and Density
The HUMBI dataset represents a significant leap in scale. Captured using a specialized 2.5-meter diameter dodecagon frame equipped with 107 synchronized GoPro HD cameras, the dataset includes 772 distinctive subjects. This setup allows for the capture of millions of images across various categories:
- Gaze: ~93K images.
- Face: ~17.3M images.
- Hand: ~24M images.
- Body: ~26M images.
Unlike many datasets that rely on professional actors in specialized suits, HUMBI captures voluntary participants in their "natural clothing." This focus on naturalism makes the data highly representative of real-world scenarios, particularly for modeling garment dynamics.
Methodology: High-Fidelity 3D Reconstruction
The technical pipeline for HUMBI involves several sophisticated steps to transform multiview 2D data into high-fidelity 3D models:
- 3D Keypoint Reconstruction: 2D detections across multiple views are triangulated into 3D keypoints for the face, hands, and body using RANSAC and non-linear refinement.
- Mesh Fitting: State-of-the-art parametric models are fitted to these keypoints, including the Surrey model for faces, MANO for hands, and SMPL for the body.
- Canonical Atlases: To handle view-specific appearance, image pixels are projected onto a 2D canonical atlas of the 3D mesh. This enables the calculation of median appearance and variance across views, highlighting how surfaces like the iris or mouth contours change based on the viewing angle.
- Garment Modeling: A shape-from-silhouette approach is used to reconstruct natural clothing, a feature often simplified in other datasets.
Key Insights and Results
The researchers found that HUMBI is highly complementary to existing benchmarks. Models trained on a combination of HUMBI and datasets like MPII-Gaze or Human3.6M consistently outperformed those trained on a single source.
Quantitative improvements were observed across all domains:
- Gaze Estimation: HUMBI achieved the smallest average bias (5.98°) compared to other major gaze datasets.
- Face Reconstruction: In monocular 3D face mesh prediction, augmenting the 3DDFA dataset with HUMBI reduced mean error from 7.1 to 4.3 pixels.
- Hand Pose: Models trained with HUMBI showed superior generalizability, with AUC improvements of up to 0.12 when tested on external datasets.
- Camera Density: Ablation studies revealed that while 107 cameras provide peak fidelity, garment reconstruction remains stable down to approximately 60 cameras.
Conclusion
HUMBI provides a massive, diverse, and high-density resource for the computer vision community. By capturing 772 subjects with 107 cameras, it enables the creation of more generalizable and realistic human models. As the dataset continues to grow through ongoing capture efforts at public venues, it will likely become a foundational tool for the next generation of digital human technologies.
Summary updated