Bundle Adjustment for Markerless Body Tracking in Monocular Video Sequences

Ali Shahrokni, Vincent Lepetit, Pascal FuaInfoscience 2003

Abstract

Researchers from CVLAB at EPFL have developed a framework for tracking 3D human motion from a single, standard video sequence without the use of markers. By adapting bundle adjustment—a technique traditionally used for rigid 3D reconstruction—to articulated structures, the method recovers motion by optimizing pose and shape parameters across multiple frames simultaneously. This approach avoids the need for expensive motion capture setups or restrictive motion priors, relying instead on image evidence and geometric constraints.

Introduction

Capturing human motion (MoCap) typically necessitates specialized environments with multiple calibrated cameras and subjects wearing reflective markers. While effective, these requirements limit MoCap's utility in real-world scenarios such as security, biomechanics, and casual animation. Monocular markerless tracking represents a significant challenge because the problem is inherently ill-posed: a single viewpoint introduces depth ambiguity, and the high dimensionality of human joint configurations creates a massive search space.

Methodology

The proposed framework utilizes a model-based bundle adjustment approach to overcome these challenges.

Articulated Body Model

The human body is represented as a collection of connected limbs modeled as cylinders and ellipsoids. Each limb is defined by specific degrees of freedom (DOF) and shape parameters, such as length and radius.

2D Feature Tracking

Rather than relying on sparse interest points, the system regularly samples points across the body's surface in a reference frame. These points are tracked across subsequent frames using intensity cross-correlation.

Overview of the markerless tracking process, showing the relationship between the 3D model and tracked 2D features across frames.

Bundle Adjustment and the Transfer Function

The core innovation lies in the simultaneous optimization of pose across a sequence of frames. The system minimizes the reprojection error—the distance between the observed 2D tracked points and the projection of the corresponding 3D points on the model.

A mathematical "transfer function" is employed to relate 2D points in a reference frame to 3D points on the articulated model. These 3D points are then projected into other frames to calculate the error.

Geometric relationship between the 3D model (cylinder), the image plane, and the camera center, illustrating the projection used to calculate error.

Anatomical Constraints

To ensure physical plausibility, the objective function includes penalty terms. These constraints prevent the optimizer from producing anatomically impossible results, such as unrealistic limb lengths, to fit noisy 2D data.

Experimental Evaluation

The method was evaluated using both synthetic and real-world data:

  • Synthetic Data: Tests on a 2D double pendulum and a 3D articulated arm demonstrated that the optimization converges to a global minimum even in the presence of Gaussian noise and outliers.
  • Real Video Sequences: The system successfully tracked a low-quality video of a golf upswing and fast arm movements on a naked torso.

The authors established a solvability requirement: the product of tracked points and frames must significantly exceed the number of unknown parameters to ensure a stable solution.

Discussion and Limitations

While the 3D optimization is robust, the system's performance is currently limited by the underlying 2D tracking. In fast-motion sequences, cumulative drift in the intensity-based correlation tracker typically caused the 3D estimation to fail after approximately 20 frames. Future work aims to incorporate more robust 2D features and address the challenges of self-occlusion, which remain a primary obstacle for single-camera tracking systems.

Summary updated

Abstract

CVLAB
BibTeX
@inproceedings{shahrokni2003bundle,
  author = {Ali Shahrokni and Vincent Lepetit and Pascal Fua},
  title = {{Bundle Adjustment for Markerless Body Tracking in Monocular Video Sequences}},
  booktitle = {Infoscience},
  year = {2003},
}
Figure