Human-M3: A Multiview Multimodal Dataset for 3d Human Pose Estimation in Outdoor Scenes

Bohao Fan, Siqi Wang, Wenxuan Guo, Wenzhao Zheng, J. H. Feng, Jie ZhouSSRN 2025 · DOI

TL;DR

This paper introduces Human-M3, a novel multi-view, multi-modal dataset for 3D human pose estimation in outdoor scenes, addressing the limitations of existing datasets that often focus on single individuals or indoor environments. They also propose a multi-modal fusion algorithm that leverages both RGB images and LiDAR point clouds to achieve more accurate 3D human pose estimation.

What Problem Does This Paper Address?

Estimating the 3D pose of humans in outdoor environments is crucial for various applications like action recognition, social behavior analysis, and scene perception. However, existing 3D human pose estimation (HPE) datasets suffer from several limitations:

  • Limited Modality: Many datasets rely on a single data type, either RGB images or point clouds, which can be insufficient for robust estimation in complex outdoor scenes.
  • Lack of Diversity: Datasets often feature only one individual per scene, failing to capture the complexities of multi-person interactions and occlusions common in real-world outdoor environments.
  • Indoor Bias: Most existing datasets are collected in controlled indoor settings, making them less suitable for training and evaluating models for outdoor scenarios where lighting, weather, and background clutter pose significant challenges.
  • Annotation Challenges: Obtaining accurate ground truth 3D poses in outdoor scenes is inherently difficult, especially for individuals not equipped with motion capture devices. Existing methods for outdoor datasets often struggle with alignment issues between 2D and 3D data or rely on motion capture, which is impractical for large-scale outdoor collection.

This paper aims to bridge this gap by introducing a comprehensive dataset and a novel approach that addresses these challenges.

Methods

The authors introduce the Human-M3 dataset, which addresses the limitations of existing datasets by:

  • Multi-view and Multi-modal: Capturing data from multiple synchronized RGB cameras and LiDAR sensors simultaneously.
  • Multi-person: Including scenes with multiple individuals, allowing for the study of interactions and occlusions.
  • Outdoor Scenes: Focusing on diverse outdoor environments like intersections, plazas, and basketball courts.

To generate accurate ground truth annotations for Human-M3, they developed a robust pipeline:

  1. Camera Extrinsic Calibration: Using LiDAR point clouds and corresponding RGB images to accurately determine the spatial relationship between cameras.
  2. Human Detection and Tracking: Employing PointPillars for pedestrian detection from point clouds and AB3DMOT for tracking.
  3. 2D Pose Estimation: Using ViTPose to obtain pixel-level accurate 2D pose estimations from RGB images.
  4. Temporal Human Pose Optimization: A novel optimization-based method that fuses 2D pose estimations with LiDAR point cloud data. This method optimizes SMPL shape and pose parameters, leveraging constraints from both modalities and temporal smoothness. The loss function includes terms for consistency with 2D HPE, point cloud alignment (chamfer loss), human shape and pose priors, and motion smoothness.

They also propose a baseline MultiModal-VoxelPose (MMVP) algorithm for multi-modal 3D HPE. MMVP integrates RGB images and point clouds by:

  • Generating 2D keypoint heatmaps from RGB images using OpenPIFPAF.
  • Transforming these 2D heatmaps into 3D heatmaps via 2D-3D projection.
  • Voxelizing and sampling the point cloud into a 3D occupancy volume.
  • Fusing these two feature sets (3D heatmaps and voxelized point clouds) and feeding them into a 3D CNN for feature extraction and 3D heatmap regression.

Datasets

  • Human-M3 Dataset (New Benchmark): This is the primary contribution of the paper. It consists of 12,200 frames captured from four different outdoor scenes, featuring multi-view RGB videos and corresponding LiDAR point clouds. It includes multiple individuals and provides 89,642 valid human pose records.
  • Comparison Datasets: The paper compares Human-M3 with existing datasets like Human3.6M, 3DPW, PedX, and LiDARH26M in terms of modality, multi-view, multi-person, and multi-scene capabilities.
  • Evaluation Datasets: The Human-M3 dataset itself is partitioned into training (90%) and testing (10%) sets for evaluation.

Key Insights and Ideas

  • The Necessity of Multi-Modality in Outdoor HPE: The paper strongly advocates for the integration of RGB and LiDAR data for robust 3D HPE in outdoor settings. Point clouds provide scale-invariant features and accurate localization, while RGB images offer rich texture information, which are complementary.
  • Challenges of Outdoor HPE: The dataset highlights significant challenges such as heavy occlusion, small object sizes due to distance, and complex interactions between multiple individuals, which are not adequately represented in existing datasets.
  • Limitations of Single-Modality Approaches: The results demonstrate that single-modality methods, while effective in some scenarios, struggle to achieve high accuracy on the Human-M3 dataset, especially compared to their performance on indoor datasets.
  • Importance of Robust Annotation: The paper emphasizes the critical role of meticulous data annotation, including manual review and temporal optimization, to ensure the quality and reliability of ground truth poses, especially in complex outdoor scenes.
  • Fusion Strategy Matters: While simple concatenation of features can improve performance, the paper suggests that more sophisticated fusion strategies can further enhance accuracy.

Key Results

  • MMVP Performance: The proposed MMVP algorithm achieved state-of-the-art results on the Human-M3 dataset, outperforming other multi-view and pointcloud-based 3D HPE algorithms.
  • MMVP achieved a Mean Per Joint Position Error (MPJPE) of 0.079m and a recall of 98.31% (at 500mm threshold).
  • It significantly improved upon other methods, for instance, achieving a 2.6cm (24%) decrease in MPJPE and at least a 0.25 increase in AP across thresholds when using multi-modal input compared to single-modal inputs.
  • Scene-Specific Performance: The MMVP algorithm showed a significant performance drop in the "intersection" scene (MPJPE of 22.62cm, recall of 77.39%), attributed to limited human pose data and frequent occlusions by vehicles.
  • Data Characteristics: Human-M3 has an average of 200 points per person, slightly more than LiDARH26M. It also exhibits a higher proportion of occluded poses compared to Panoptic, making it a more challenging benchmark.

Limitations and Future Work

  • Dataset Limitations:
    • The limited range of scenes and human motion patterns covered in Human-M3 may not fully represent the diversity of real-world scenarios.
  • The "intersection" scene poses a particular challenge due to data distribution disparities.
  • Algorithm Limitations:
    • The proposed optimization algorithm, while efficient, did not integrate advanced techniques like VAE.
  • Manual intervention and screening are still necessary for annotation, which is time-consuming.
  • Future Work:
    • Further investigation into more advanced fusion techniques for multi-modal data.
  • Implementation of temporal learning approaches to improve labeling accuracy and leverage stronger motion priors.
  • More comprehensive data collection strategies to cover a wider range of scenarios and human motion patterns.
  • Exploring more advanced fusion techniques to unlock the full potential of multi-modal datasets.

Summary updated

BibTeX
@misc{fan2025human,
  author = {Bohao Fan and Siqi Wang and Wenxuan Guo and Wenzhao Zheng and J. H. Feng and Jie Zhou},
  title = {{Human-M3: A Multiview Multimodal Dataset for 3d Human Pose Estimation in Outdoor Scenes}},
  year = {2025},
  doi = {10.2139/ssrn.5144790},
  howpublished = {SSRN},
}
Figure