EasyRet3D: Uncalibrated Multi-View Multi-Human 3D Reconstruction and Tracking

Junjie Oscar Yin, Ting Li, Jiahao Wang, Yi Zhang, Alan L. YuilleWACV 2025 · DOI

Introduction

Capturing human motion in 3D from multiple viewpoints is a cornerstone of computer vision, yet it remains tethered to the cumbersome requirement of camera calibration. Traditional methods necessitate precise knowledge of camera extrinsics (position and orientation) and intrinsics (lens properties), often requiring manual intervention or specialized patterns like checkerboards. EasyRet3D introduces a paradigm shift by enabling high-fidelity 3D reconstruction and tracking from uncalibrated, "in-the-wild" video streams.

Overview of the EasyRet3D framework showing uncalibrated input videos and the resulting 3D human tracking and reconstruction.

The Calibration Challenge

The primary bottleneck in multi-view 3D reconstruction is the "calibration headache." In scenarios like social media footage or spontaneous security setups, manual calibration is impossible. Furthermore, existing end-to-end neural networks often fail to generalize outside their training datasets. Optimization-based methods, while more flexible, are notoriously slow—sometimes requiring hours to process less than a minute of video.

The EasyRet3D Framework

EasyRet3D addresses these issues through a modular, four-stage pipeline that leverages the geometric consistency of human subjects themselves.

  1. Single-view Initialization: The system utilizes foundation models (e.g., PHALP and HMR2.0) to detect individuals and estimate initial 3D poses within each camera's local coordinate system.
  2. Automatic Calibration: By treating human detections as "geometric anchors" and matching individuals across views using 3D appearance embeddings, the system solves for camera positions. This approach assumes a common ground plane and employs a RANSAC-based Perspective-n-Point (PnP) solver.
  3. Adaptive Stitching: 3D tracklets from disparate cameras are merged into a global world coordinate system. The module intelligently weights data based on 2D keypoint confidence, prioritizing clear views over occluded ones.
  4. Iterative Global Optimization: To overcome the computational lag of traditional motion models, EasyRet3D employs a Neural Motion Field (NeMF). This allows for parallelized optimization across the entire video sequence, achieving a 30x speedup over previous state-of-the-art methods.

Experimental Results

The framework was rigorously tested on major benchmarks, including the Panoptic dome dataset, EgoHumans, Shelf, and Human3.6M.

Detailed 3D reconstruction of multiple subjects in a social interaction scenario.
  • State-of-the-Art Accuracy: EasyRet3D achieved a PA-MPJPE of 25.7mm on Human3.6M, the lowest error reported for a calibration-free method.
  • Robust Tracking: On the Panoptic dataset, the system reached a Multiple Object Tracking Accuracy (MOTA) of 98.9 with zero identity switches, demonstrating exceptional resilience to occlusions.
  • Efficiency: The transition to NeMF-based optimization reduced processing times from hours to minutes, making the framework viable for practical applications.

Conclusion and Future Work

EasyRet3D demonstrates that humans can serve as effective calibration targets, eliminating the need for pre-defined patterns. While currently limited to static camera setups, the framework's modular nature allows for future integration with SLAM for moving cameras and adaptation to non-human subjects by swapping underlying mesh models. By combining geometric priors with modern neural optimization, EasyRet3D paves the way for scalable 3D human analysis in unconstrained environments.

Summary updated

BibTeX
@inproceedings{yin2025easyret,
  author = {Junjie Oscar Yin and Ting Li and Jiahao Wang and Yi Zhang and Alan L. Yuille},
  title = {{EasyRet3D: Uncalibrated Multi-View Multi-Human 3D Reconstruction and Tracking}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00309},
}
Figure