SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments

Yudi Dai, Yitai Lin, Xiping Lin, Chenglu Wen, Lan Xu, Hongwei Yi, Siqi Shen, Yuexin Ma, Cheng Wang

Abstract

SLOPER4D is the first large-scale, scene-aware dataset for Global 4D Human Pose Estimation (GHPE) in urban environments. By fusing LiDAR, cameras, and body-worn IMUs, the dataset provides high-fidelity 3D human motion data accurately localized within complex city scenes across trajectories up to 1.3km. This work addresses the limitations of existing datasets that are either restricted to indoor labs or lack global context, setting a new benchmark for "in-the-wild" human-scene interaction.

Introduction

While 3D Human Pose Estimation (HPE) has made significant progress, most research remains confined to small-scale indoor settings or "in-the-wild" datasets that lack global context. Estimating a person's pose relative to their own root is common, but accurately placing them in a city-scale coordinate system or understanding their interaction with the physical environment—such as walking on uneven pavement or leaning against a wall—remains a challenge.

Existing datasets often fall into two categories: marker-based systems restricted to laboratory environments, or monocular video datasets that lack depth and suffer from trajectory drift over long distances. To enable applications like autonomous driving or augmented reality in urban spaces, a dataset providing both fine-grained pose and accurate global positioning across hundreds of meters is essential.

Methodology

Hardware Setup

The authors developed a unique "follow-the-leader" hardware configuration to capture multi-modal data. The system consists of two modules: a "scanner" and a "performer."

The hardware setup includes a head-mounted sensor suite (LiDAR and camera) and a body-worn IMU system for the performer.
  • Scanner: Wears a helmet equipped with a high-beam LiDAR (Ouster-Os1-128) and a wide-angle camera (DJI-Action2).
  • Performer: Wears 17 wireless IMUs to capture local joint movements.

Joint Optimization Pipeline

To transform raw multi-modal data into high-quality ground truth, a joint optimization pipeline was proposed. This pipeline integrates LiDAR SLAM for building globally consistent 3D maps with IMU-based pose estimation.

The construction pipeline involves multi-sensor synchronization and joint optimization, incorporating mesh-to-points, motion smoothness, and scene-aware contact terms.

The optimization minimizes a complex loss function including:

  • Mesh-to-Points Term: Aligning the SMPL human mesh with the LiDAR point cloud.
  • Scene-Aware Contact Term: Ensuring feet maintain realistic contact with the ground, preventing "floating" or "sinking" artifacts.
  • Smoothness and Pose Priors: Maintaining physical plausibility and preventing temporal jitter.

The SLOPER4D Dataset

SLOPER4D stands out due to its scale and diversity compared to predecessors like 3DPW or Human3.6M:

  • Scope: 12 subjects across 10 diverse urban scenes, including commercial streets, parks, and sports fields.
  • Scale: 15 sequences with trajectories exceeding 200m (maximum 1.3km) and areas up to 13,000 $m^2$.
  • Data Volume: 100k LiDAR frames, 300k video frames, and 500k IMU frames.
  • Annotations: Includes 2D keypoints, 3D SMPL parameters, global translations, and reconstructed 3D scene meshes.

Key Insights and Results

  • LiDAR as a Global Anchor: While IMUs are effective for local joint movement, they drift over time. LiDAR provides the geometric constraints necessary to anchor the human model to the global map.
  • Scene Constraints: Standard ICP often fails on sparse human point clouds. The addition of a "scene-aware contact term" ensures poses are physically plausible within the reconstructed environment.
  • Benchmarking: Evaluations revealed a significant domain gap; models trained on existing datasets perform poorly on SLOPER4D's urban-scale challenges. Furthermore, LiDAR-based methods achieved 13% lower trajectory error than image-only methods.

Conclusion

SLOPER4D provides a challenging and necessary benchmark for the field of global human pose estimation. While currently focused on single-person trajectories, future work aims to address multi-person scenarios, online sensor synchronization, and the integration of camera textures for more realistic scene-human integration.

Summary updated

Abstract

We present SLOPER4D, a novel scene-aware dataset collected in large urban environments to facilitate the research of global human pose estimation (GHPE) with human-scene interaction in the wild. Employing a head-mounted device integrated with a LiDAR and camera, we record 12 human subjects' activities over 10 diverse urban scenes from an egocentric view. Frame-wise annotations for 2D key points, 3D pose parameters, and global translations are provided, together with reconstructed scene point clouds. To obtain accurate 3D ground truth in such large dynamic scenes, we propose a joint optimization method to fit local SMPL meshes to the scene and fine-tune the camera calibration during dynamic motions frame by frame, resulting in plausible and scene-natural 3D human poses. Even-tually, SLOPER4D consists of 15 sequences of human motions, each of which has a trajectory length of more than 200 meters (up to 1,300 meters) and covers an area of more than 200 m 2 (up to 30,000 m 2 ), including more than 100k LiDAR frames, 300k video frames, and 500k IMU-based motion frames. With SLOPER4D, we provide a detailed and thorough analysis of two critical tasks, including camera-based 3D HPE and LiDAR-based 3D HPE in urban environments, and benchmark a new task, GHPE. The in-depth analysis demonstrates SLOPER4D poses significant challenges to existing methods and produces great research opportunities. The dataset and code are released at http://www.lidarhumanmotion.net/sloper4d/.
BibTeX
@inproceedings{dai2023sloper,
  author = {Yudi Dai and Yitai Lin and Xiping Lin and Chenglu Wen and Lan Xu and Hongwei Yi and Siqi Shen and Yuexin Ma and Cheng Wang},
  title = {{SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure