Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recovery

Fengyuan Yang, Kerui Gu, Ha Linh Nguyen, Angela Yao, Yao, Angela

Abstract

Accurate camera motion estimation is essential for recovering global human motion in world coordinates from RGB video inputs. SLAM is widely used for estimating camera trajectory and point cloud, but monocular SLAM does so only up to an unknown scale factor. Previous works estimate the scale factor through optimization, but this is unreliable and time-consuming. This paper presents an optimization-free scale calibration framework, Human as Checkerboard (HAC). HAC innovatively leverages the human body predicted by human mesh recovery models as a calibration reference. Specifically, it uses the absolute depth of human-scene contact joints as references to calibrate the corresponding relative scene depth from SLAM. HAC benefits from geometric priors encoded in human mesh recovery models to estimate the SLAM scale and achieves precise global human motion estimation. Simple yet powerful, our method sets a new state-of-the-art performance for global human mesh estimation tasks, reducing motion errors by 50% over prior local-to-global methods while using 100x less inference time than optimization-based methods.

Introduction: The Scale Ambiguity Problem

Recovering 3D human motion in "world coordinates"—knowing exactly how many meters someone traveled across a room—is a major challenge in computer vision. Most Human Mesh Recovery (HMR) models operate in "camera coordinates," meaning they only determine the person's position relative to the lens.

To achieve global coordinates, we must account for camera movement. While monocular SLAM (Simultaneous Localization and Mapping) is highly effective at tracking camera trajectories, it suffers from scale ambiguity: it cannot distinguish between a camera moving one meter in a real room or one centimeter in a miniature model. Existing solutions often rely on "black-box" models that guess global motion from local poses—which fails in scenarios like walking on a treadmill—or heavy optimization processes that can take hours to process a single video.

Methods: Humans as Checkerboards

The core innovation of HAC is a clever analogy: just as traditional robotics uses checkerboards of known dimensions to calibrate cameras, HAC proposes that the human body itself can serve as a known calibration reference.

The HAC Framework

The system employs a dual-stream architecture:

  1. Parallel Processing: A monocular SLAM system (e.g., DroidSLAM) estimates camera motion and a scene point cloud at an arbitrary scale. Simultaneously, a local HMR model predicts the human mesh and its absolute depth in camera coordinates.
  2. Scale Calibration via Contacts: The system identifies human-scene contact points, typically the feet. By finding the lowest joint on the human mesh and its corresponding point in the SLAM point cloud, the system can link the two coordinate systems.
  3. Calculating the Scale Factor: By comparing the absolute depth from the HMR model ($d^A$) to the relative depth from the SLAM system ($d^S$), the system calculates a scale factor ($s = d^A / d^S$).
  4. Robust Aggregation: To mitigate noise, HAC calculates the median scale factor across the entire video sequence, ensuring a stable global trajectory.
HAC recovers global human motion by combining camera motion from SLAM with local human motion, effectively treating the human as a calibration reference.

Handling Occlusions

In cases where contact joints are not visible, such as in egocentric footage from head-mounted cameras, HAC utilizes a ground-plane fallback. It uses the predicted human mesh to guide a ground-plane fitting algorithm, assuming the subject is standing on a floor, to estimate the missing depth information.

Key Results and Evaluation

The authors evaluated HAC on the EMDB (Electromagnetic Database) and EgoBody datasets. EMDB provides high-quality ground truth for global motion, while EgoBody tests the system's robustness in challenging egocentric views.

Performance and Efficiency

HAC achieves state-of-the-art accuracy, reducing global motion errors by 50% compared to previous local-to-global methods like WHAM. Crucially, it achieves this without heavy optimization, making it 100x faster than optimization-based methods like SLAHMR. It can process 1,000 frames in approximately 5 seconds on a standard GPU.

Quantitative comparison showing that HAC (Ours) achieves trajectories closer to ground truth (GT) while being significantly faster than optimization-based methods.

Versatility

The method demonstrated superior performance across various HMR backbones (CLIFF, HMR2.0, VIMO) and SLAM systems (DroidSLAM, DPVO), proving its effectiveness as a versatile, "plug-and-play" framework.

Discussion and Future Work

HAC demonstrates that modern HMR models are "good enough" for calibration. While human sizes vary, the depth error of current models (~0.1m) is significantly smaller than the typical global trajectory errors (>1m) of other methods.

However, HAC remains dependent on the underlying SLAM system; it may struggle in featureless environments where SLAM fails. Future work could explore multi-person scenarios, using multiple individuals as simultaneous "checkerboards" to further enhance scale robustness and resolve long-term drift in extended sequences.

Project page: https://martayang.github.io/HAC

Summary updated

Abstract

Accurate camera motion estimation is essential for recovering global human motion in world coordinates from RGB video inputs. SLAM is widely used for estimating camera trajectory and point cloud, but monocular SLAM does so only up to an unknown scale factor. Previous works estimate the scale factor through optimization, but this is unreliable and time-consuming. This paper presents an optimization-free scale calibration framework, Human as Checkerboard (HAC). HAC innovatively leverages the human body predicted by human mesh recovery model as a calibration reference. Specifically, it uses the absolute depth of human-scene contact joints as references to calibrate the corresponding relative scene depth from SLAM. HAC benefits from geometric priors encoded in human mesh recovery models to estimate the SLAM scale and achieves precise global human motion estimation. Simple yet powerful, our method sets a new state-of-the-art performance for global human mesh estimation tasks, reducing motion errors by 50% over prior local-to-global methods while using 100$\times$ less inference time than optimization-based methods. Project page: https://martayang.github.io/HAC.
BibTeX
@misc{yang2024humans,
  author = {Fengyuan Yang and Kerui Gu and Ha Linh Nguyen and Angela Yao and Yao, Angela},
  title = {{Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recovery}},
  year = {2024},
  eprint = {2407.00574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.00574},
}
Figure