Abstract
SLOPER4D is the first large-scale, scene-aware dataset for Global 4D Human Pose Estimation (GHPE) in urban environments. By fusing LiDAR, cameras, and body-worn IMUs, the dataset provides high-fidelity 3D human motion data accurately localized within complex city scenes across trajectories up to 1.3km. This work addresses the limitations of existing datasets that are either restricted to indoor labs or lack global context, setting a new benchmark for "in-the-wild" human-scene interaction.
Introduction
While 3D Human Pose Estimation (HPE) has made significant progress, most research remains confined to small-scale indoor settings or "in-the-wild" datasets that lack global context. Estimating a person's pose relative to their own root is common, but accurately placing them in a city-scale coordinate system or understanding their interaction with the physical environment—such as walking on uneven pavement or leaning against a wall—remains a challenge.
Existing datasets often fall into two categories: marker-based systems restricted to laboratory environments, or monocular video datasets that lack depth and suffer from trajectory drift over long distances. To enable applications like autonomous driving or augmented reality in urban spaces, a dataset providing both fine-grained pose and accurate global positioning across hundreds of meters is essential.
Methodology
Hardware Setup
The authors developed a unique "follow-the-leader" hardware configuration to capture multi-modal data. The system consists of two modules: a "scanner" and a "performer."
- Scanner: Wears a helmet equipped with a high-beam LiDAR (Ouster-Os1-128) and a wide-angle camera (DJI-Action2).
- Performer: Wears 17 wireless IMUs to capture local joint movements.
Joint Optimization Pipeline
To transform raw multi-modal data into high-quality ground truth, a joint optimization pipeline was proposed. This pipeline integrates LiDAR SLAM for building globally consistent 3D maps with IMU-based pose estimation.
The optimization minimizes a complex loss function including:
- Mesh-to-Points Term: Aligning the SMPL human mesh with the LiDAR point cloud.
- Scene-Aware Contact Term: Ensuring feet maintain realistic contact with the ground, preventing "floating" or "sinking" artifacts.
- Smoothness and Pose Priors: Maintaining physical plausibility and preventing temporal jitter.
The SLOPER4D Dataset
SLOPER4D stands out due to its scale and diversity compared to predecessors like 3DPW or Human3.6M:
- Scope: 12 subjects across 10 diverse urban scenes, including commercial streets, parks, and sports fields.
- Scale: 15 sequences with trajectories exceeding 200m (maximum 1.3km) and areas up to 13,000 $m^2$.
- Data Volume: 100k LiDAR frames, 300k video frames, and 500k IMU frames.
- Annotations: Includes 2D keypoints, 3D SMPL parameters, global translations, and reconstructed 3D scene meshes.
Key Insights and Results
- LiDAR as a Global Anchor: While IMUs are effective for local joint movement, they drift over time. LiDAR provides the geometric constraints necessary to anchor the human model to the global map.
- Scene Constraints: Standard ICP often fails on sparse human point clouds. The addition of a "scene-aware contact term" ensures poses are physically plausible within the reconstructed environment.
- Benchmarking: Evaluations revealed a significant domain gap; models trained on existing datasets perform poorly on SLOPER4D's urban-scale challenges. Furthermore, LiDAR-based methods achieved 13% lower trajectory error than image-only methods.
Conclusion
SLOPER4D provides a challenging and necessary benchmark for the field of global human pose estimation. While currently focused on single-person trajectories, future work aims to address multi-person scenarios, online sensor synchronization, and the integration of camera textures for more realistic scene-human integration.
Summary updated