HUMBI 1.0: HUman Multiview Behavioral Imaging Dataset

Zhixuan Yu, Jae Shin Yoon, Prashanth Venkatesh, Jaesik Park, Jihun Yu, Hyun Soo Park

Abstract

This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and physical condition. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance using their canonical atlas. We demonstrate that HUMBI is highly effective in learning and reconstructing a complete human model and is complementary to the existing datasets of human body expressions with limited views and subjects such as MPII-Gaze, Multi-PIE, Human3.6M, and Panoptic Studio datasets.

Introduction: Bridging the Dataset Gap

Creating photorealistic 3D digital humans remains a significant challenge in computer vision. To achieve high fidelity, models must understand how appearance—from the glint in an eye to the folds in a shirt—changes based on the viewpoint. Historically, researchers have faced a stark trade-off: datasets either offered high subject diversity with very few camera angles (e.g., MPII-Gaze) or a dense array of cameras with very few subjects (e.g., CMU Panoptic Studio). This lack of diversity hinders the ability of AI models to generalize across different ages, ethnicities, and clothing styles.

HUMBI occupies a unique space in the dataset landscape, offering both high subject diversity and a high number of viewpoints.

HUMBI (HUman Multiview Behavioral Imaging) addresses this "viewpoint vs. subject" gap. By capturing 772 diverse subjects using a dense array of 107 synchronized cameras, it provides the necessary data to learn both complex geometry and view-specific appearance.

Methodology and Reconstruction Pipeline

The HUMBI capture stage consists of a dodecagon frame equipped with 107 synchronized GoPro HD cameras. To transform these 2D streams into usable 3D data, the authors developed a sophisticated reconstruction pipeline:

  • 3D Keypoints: Joints and facial features are located in 3D space using triangulation and RANSAC.
  • Parametric Meshes: Industry-standard models are fitted to the data, including Surrey for faces, MANO for hands, and SMPL for the body.
  • Canonical Atlases: By mapping multiview images onto a flattened 2D "canonical atlas" of the 3D surface, the system tracks how appearance varies by viewpoint at every point on the body.
  • Garment Modeling: Unlike many datasets that treat clothing as part of the body volume, HUMBI explicitly models garments as separate layers over the body mesh.
The HUMBI dataset provides high-fidelity 3D reconstructions for gaze, face, hands, body, and garments across diverse subjects.

Key Insights: Diversity and Complementarity

A core finding is that HUMBI is highly complementary to existing datasets. When models are trained on standard datasets augmented with HUMBI, performance improves significantly. This is because HUMBI fills the "viewpoint gap" inherent in single-view or limited-view data.

Furthermore, the dataset captures "natural" diversity. Because subjects were voluntary participants at public events wearing their own clothing, HUMBI includes a wide range of real-world fabrics, wrinkles, and body shapes that are often missing from lab-controlled environments. The authors also quantified appearance variance, finding that features like the iris and nose exhibit high variance across angles, emphasizing the need for multiview data.

Results

Quantitative evaluations demonstrate HUMBI's effectiveness across multiple tasks:

  • Face Reconstruction: Adding HUMBI to the 3DDFA dataset reduced mean pixel error from 7.1 to 4.3.
  • Gaze Estimation: HUMBI achieved a remarkably low average bias of 5.98° compared to other major gaze datasets.
  • Body Mesh Prediction: On the UP-3D benchmark, models augmented with HUMBI improved mesh prediction accuracy by nearly 20% (reducing error from 22.7 to 18.4 pixels).
  • Ablation Studies: Analysis suggests that while 107 cameras provide the highest fidelity, high-quality garment reconstruction can be achieved with approximately 60 cameras.

Conclusion and Future Work

While HUMBI is extensive, it currently lacks the extreme athletic poses found in specialized motion-capture datasets. However, HUMBI is designed as a living dataset. With ongoing data collection at public venues, it is expected to grow in subject diversity and pose complexity. The ultimate goal is to leverage this data to move from static 3D models toward authentic, real-time telepresence.

Summary updated

Abstract

This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and physical condition. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance using their canonical atlas. We demonstrate that HUMBI is highly effective in learning and reconstructing a complete human model and is complementary to the existing datasets of human body expressions with limited views and subjects such as MPII-Gaze, Multi-PIE, Human3.6M, and Panoptic Studio datasets.
BibTeX
@misc{yu2018humbi,
  author = {Zhixuan Yu and Jae Shin Yoon and Prashanth Venkatesh and Jaesik Park and Jihun Yu and Hyun Soo Park},
  title = {{HUMBI 1.0: HUman Multiview Behavioral Imaging Dataset}},
  year = {2018},
  eprint = {1812.00281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00281},
}
Figure