EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild

Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan José Zarate, Otmar Hilliges

Introduction

Accurate 3D human pose and shape (HPS) estimation is a cornerstone of computer vision, with applications ranging from augmented reality to robotics. However, most existing research focuses on camera-relative poses—tracking limb movement relative to the torso. While sufficient for some tasks, this approach fails to capture "global" trajectories: the movement of a person within a world coordinate system, especially when the camera itself is in motion.

The Electromagnetic Database (EMDB) addresses this gap. By leveraging wearable electromagnetic (EM) sensors and a sophisticated neural optimization pipeline, EMDB provides high-accuracy, drift-free global trajectories in diverse, "in-the-wild" environments.

The Challenge of Global Capture

Capturing ground-truth global motion is traditionally a trade-off between accuracy and mobility. Indoor optical systems like Vicon offer millimeter-level precision but are confined to specialized labs. Conversely, outdoor datasets often rely on Inertial Measurement Units (IMUs). While portable, IMUs suffer from "drift"—accumulated errors over time that make long-range tracking unreliable.

EMDB overcomes these limitations by using EM sensors. Unlike IMUs, which measure orientation, EM sensors provide both position and orientation relative to a base station. This allows for precise local tracking without the line-of-sight requirements of optical systems, making it ideal for occluded outdoor environments.

The EMP Pipeline: From Sensors to Pixels

To generate the dataset, the researchers developed the Electromagnetic Poser (EMP), a three-stage optimization pipeline designed to produce lab-quality data in unconstrained settings:

  1. Stage 1: Local EM Pose: Participants wear 12 wireless EM sensors. The system fits the SMPL (Skinned Multi-Person Linear) body model to these sensor readings relative to a base station worn on the lower back.
  2. Stage 2: World Alignment: The local pose is integrated into a global coordinate frame. This stage jointly optimizes EM data, 2D keypoints from video, depth data, and the camera's own trajectory (tracked via iPhone ARKit).
  3. Stage 3: Pixel-Level Refinement: To eliminate temporal jitter and improve visual alignment, the authors employ a neural implicit avatar model (based on Vid2Avatar). By reconstructing a personalized 3D model and comparing its rendered appearance to the video pixels, the system "nudges" the pose into perfect alignment with the image.

Dataset Characteristics

EMDB represents a significant leap in "in-the-wild" data quality and scope:

  • Volume: 58 minutes of data across 105,000 frames and 81 sequences.
  • Diversity: 10 participants (5 male, 5 female) performing complex movements in various indoor and outdoor settings.
  • Annotations: Includes high-quality SMPL parameters, global body trajectories, and global camera trajectories.

The methodology was rigorously validated using an indoor Multi-view Volumetric Capture (MVS) system with 53 cameras, ensuring the "in-the-wild" results are grounded in lab-quality truth.

Key Results and Benchmarking

EMDB establishes a new standard for accuracy in unconstrained environments:

  • Precision: Validation showed a positional error of only 2.3 cm and an angular error of 10.6 degrees, outperforming the previous gold standard for in-the-wild data, 3DPW (which reports 2.6 cm and 12.1 degrees).
  • Drift Resistance: In long-range tests (112-meter walks), the system exhibited a drift of only 0.7%, proving its robustness for extended tracking.
  • A New Benchmark: When state-of-the-art monocular RGB methods (e.g., HybrIK, ROMP) were tested on EMDB, their performance dropped significantly compared to lab-based benchmarks. This confirms that EMDB serves as a necessary "stress test" for the community, highlighting the difficulty of global pose estimation in the real world.

Limitations and Future Directions

While EMDB is a landmark dataset, it has limitations. It currently focuses on single-person scenarios; multi-person interactions in-the-wild remain an open challenge. Additionally, EM sensors are sensitive to large metal structures, which can distort magnetic fields in certain indoor environments. Finally, the global accuracy remains partially dependent on the quality of the handheld camera's tracking (ARKit).

Conclusion

By bridging the gap between lab-quality precision and real-world complexity, EMDB provides the foundation for the next generation of human motion models. It challenges the field to move beyond camera-relative poses and toward a true understanding of human movement in the global 3D world.

Summary updated

Abstract

We present EMDB, the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos. We use body-worn, wireless electromagnetic (EM) sensors and a hand-held iPhone to record a total of 58 minutes of motion data, distributed over 81 indoor and outdoor sequences and 10 participants. Together with accurate body poses and shapes, we also provide global camera poses and body root trajectories. To construct EMDB, we propose a multi-stage optimization procedure, which first fits SMPL to the 6-DoF EM measurements and then refines the poses via image observations. To achieve high-quality results, we leverage a neural implicit avatar model to reconstruct detailed human surface geometry and appearance, which allows for improved alignment and smoothness via a dense pixel-level objective. Our evaluations, conducted with a multi-view volumetric capture system, indicate that EMDB has an expected accuracy of 2.3 cm positional and 10.6 degrees angular error, surpassing the accuracy of previous in-the-wild datasets. We evaluate existing state-of-the-art monocular RGB methods for camera-relative and global pose estimation on EMDB. EMDB is publicly available under https://ait.ethz.ch/emdb.
BibTeX
@inproceedings{kaufmann2023emdb,
  author = {Manuel Kaufmann and Jie Song and Chen Guo and Kaiyue Shen and Tianjian Jiang and Chengcheng Tang and Juan José Zarate and Otmar Hilliges},
  title = {{EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01345},
}
Figure