EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

Manuel Kaufmann, Yi Zhao, Chengcheng Tang, Lingling Tao, Christopher D. Twigg, Jie Song, Robert Wang, Otmar Hilliges

EM-POSE is a novel framework for reconstructing full-body 3D human poses using a sparse set of 6–12 wireless electromagnetic (EM) sensors. By combining a custom wearable EM tracking system with a "Learned Gradient Descent" (LGD) algorithm, the researchers achieved high-accuracy tracking that is entirely immune to the visual occlusions that plague camera-based systems and the drift that affects IMUs.

What Problem Does This Paper Address?

Achieving "full-body presence" in AR/VR is a major hurdle. Current solutions generally fall into two categories, both with significant drawbacks:

  1. Optical Systems: External cameras (like Optitrack) limit mobility to a specific room, while head-mounted cameras (like those on a Quest headset) suffer from self-occlusion—for example, when arms are behind the back or when the user is crouching.
  2. Inertial Measurement Units (IMUs): While wearable and occlusion-proof, IMUs only measure relative motion (acceleration and angular velocity). Over time, small errors accumulate, leading to "drift" where the virtual avatar's limbs slowly slide away from the user's actual position.

The authors propose using Electromagnetic (EM) sensing. An EM source worn on the body (e.g., the lower back) creates a magnetic field, and small sensors measure their absolute 6D pose (position and orientation) relative to that source. This provides a wearable, occlusion-proof solution with absolute positioning that doesn't drift.

Overview of the EM-POSE system: (A) The occlusion problem in external views, (B) the EM sensing setup, and (C) the pose estimation pipeline.

Methods

The researchers developed a custom hardware and software stack to make EM-based tracking viable for human motion:

  • Custom Wireless Hardware: They miniaturized 3-axis sensing coils into low-power, wireless sensors (roughly 8 x 3.8 x 0.8 cm) that update at 120Hz.
  • Learned Gradient Descent (LGD): This is the core of the software. Instead of using a standard mathematical optimizer to fit a body model (SMPL) to the sensor data—which can be slow and prone to "unnatural" poses—they trained a neural network to predict the optimization steps. This hybrid approach ensures the resulting pose stays on the "manifold" of valid human movements.
  • Bridging the Sim-to-Real Gap: Since capturing enough real EM data to train a deep network from scratch is impractical, the authors used the AMASS dataset to simulate virtual sensors. Crucially, they modeled "skin-to-sensor" offsets—the fact that a sensor sits on top of clothes/skin rather than directly on the bone—to make the synthetic training data match real-world conditions.
Hardware components including the 3D sensor coil, measurement platform (PCB), and the 3D source coil.

Datasets

  • AMASS: Used for large-scale synthetic training by simulating virtual EM sensors on thousands of human motion sequences.
  • New EM-Pose Dataset: The authors introduced a new benchmark consisting of 45 sequences (~36.8 minutes) from 5 subjects. Ground truth was established using a complex multi-view setup involving 4 RGB-D cameras and an Optitrack system. This dataset is publicly available to the research community.

Key Insights and Ideas

  • 6D is better than 3D: Unlike many previous attempts that only used sensor positions, EM-POSE leverages the full 6D pose (position + orientation). The authors found that modeling rotational offsets is actually more critical for accuracy than translational offsets.
  • The Power of Priors: By using the SMPL parametric body model, the system can "fill in the blanks." Even with only 6 sensors (wrists, lower legs, head, and back), the system can plausibly estimate the positions of the elbows and knees.
  • Occlusion Immunity: Because magnetic fields pass through the human body, the system maintains high accuracy during complex movements like crossing arms or crouching—scenarios where computer vision systems typically fail.

Key Results

The system demonstrates state-of-the-art performance for sparse wearable tracking:

  • Accuracy: With 12 sensors, the mean per-joint positional error (MPJPE) was just 31.8 mm. Even stripping down to a minimal 6-sensor setup, the error only increased slightly to 35.4 mm.
  • Comparison: EM-POSE significantly outperformed traditional optimization methods (like MoSh++) and pure deep learning baselines (like BiRNNs), particularly in its ability to generalize to new subjects.
  • Speed: The LGD framework allows for fast convergence, making the approach a candidate for real-time applications.

Limitations and Future Work

  • Magnetic Interference: The biggest challenge for this system is metal. Large metallic objects or other electronics within 1.5 meters can distort the magnetic field, leading to tracking errors.
  • Shape Estimation: While the system is excellent at pose (how you are moving), it struggles to capture detailed body shape (e.g., individual muscle mass or body fat) from just 6–12 points on the skin.
  • Hardware Robustness: The authors noted that EM data can occasionally be noisy or drop frames, suggesting that future iterations could benefit from even more robust signal processing or sensor fusion (e.g., combining EM with IMUs).

Summary updated

Abstract

Fully immersive experiences in AR/VR depend on re-constructing the full body pose of the user without restricting their motion. In this paper we study the use of body-worn electromagnetic (EM) field-based sensing for the task of 3D human pose reconstruction. To this end, we present a method to estimate SMPL parameters from 6-12 EM sensors. We leverage a customized wearable system consisting of wireless EM sensors measuring time-synchronized 6D poses at 120 Hz. To provide accurate poses even with little user instrumentation, we adopt a recently proposed hybrid framework, learned gradient descent (LGD), to iteratively estimate SMPL pose and shape from our input measurements. This allows us to harness powerful pose priors to cope with the idiosyncrasies of the input data and achieve accurate pose estimates. The proposed method uses AMASS to synthesize virtual EM-sensor data and we show that it generalizes well to a newly captured real dataset consisting of a total of 36 minutes of motion from 5 subjects. We achieve reconstruction errors as low as 31.8 mm and 13.3 degrees, outperforming both pure learning- and pure optimization-based methods. Code and data is available under https://ait.ethz.ch/projects/2021/em-pose.
BibTeX
@inproceedings{kaufmann2021em,
  author = {Manuel Kaufmann and Yi Zhao and Chengcheng Tang and Lingling Tao and Christopher D. Twigg and Jie Song and Robert Wang and Otmar Hilliges},
  title = {{EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01131},
}
Figure