Full-motion recovery from multiple video cameras applied to face tracking and recognition

Josh Harguess, Changbo Hu, J. K. AggarwalWorkshops @ ICCV 2011 · DOI

Abstract

Robust 3D motion recovery of objects, such as human faces, is a critical challenge in computer vision, particularly under large rotations where single-camera views often fail. This paper presents a method for full 6-DOF motion recovery by explicitly fusing data from multiple overlapping video cameras into a joint calculation. By modeling the head as a 3D cylinder and combining spatial and temporal gradients from all views simultaneously, the proposed system achieves significantly higher pose estimation accuracy and improved face recognition rates compared to independent single-camera tracking.

Introduction

Tracking an object in 3D space—recovering its three rotations and three translations—is fundamental to many vision tasks. A major hurdle in face tracking is the "nonfrontal pose" problem: significant head turns cause a single camera to lose key features, leading to tracking failure. While multi-camera environments are increasingly common, typical systems process each view independently and attempt to merge results post-hoc. This research argues for a more integrated approach: fusing raw information at the motion-calculation level to maintain a continuous, accurate 3D track even when individual cameras have suboptimal views.

Methodology

The authors utilize a model-based approach, representing the human head as a 3D cylinder. The technical core is an extension of the Lucas-Kanade image alignment algorithm into a multi-camera framework.

Coordinate Synchronization and Joint Calculation

The system defines a primary "world" coordinate system. The relative positions and orientations of all other cameras are pre-calculated using transformation matrices ($C_i$). Rather than calculating motion per camera, the system concatenates spatial and temporal image gradients from all cameras into a single, global system of equations.

Conceptual diagram of 3D motion recovery and multi-camera coordinate synchronization.

Motion Representation and Texture Mapping

To handle 3D rotations, a "twist" approximation is used for small-angle changes, linearizing the complex motion into a solvable mathematical format ($\Delta\mu$). Once the 3D motion is recovered, the image data is "unwrapped" from the cylindrical model to create a normalized, frontal-looking texture map. This map serves as a standardized input for face recognition algorithms, effectively neutralizing the impact of the subject's pose.

Experimental Evaluation

The system was evaluated on two primary datasets:

  1. Pose Estimation: A video sequence of a subject wearing a checkerboard pattern, providing "ground truth" 3D rotation and translation via standard calibration tools.
  2. Face Recognition: A dataset of 18 subjects (100 frames each) captured simultaneously by two cameras.

Results and Discussion

The multi-camera approach demonstrated substantial improvements in both stability and accuracy over single-camera setups.

Pose Accuracy

Quantitative analysis revealed that fusing gradients from multiple cameras significantly reduces estimation error. For instance, in yaw/pan estimation, the Mean Absolute Error (MAE) was reduced from 13.9 degrees in a single-camera setup to just 3.8 degrees with the multi-camera fusion.

Comparison of pan angle estimation accuracy: the multi-camera approach (red) closely tracks the ground truth (black), while the single-camera model (blue) deviates significantly.

Tracking Stability and Recognition

Single-camera models frequently "lost" the face during sharp turns. In contrast, the multi-camera model remained stable by leveraging overlapping views; when one camera's view became occluded or highly oblique, the other camera's gradients compensated. This improved tracking directly translated to better recognition. Using the Eigenfaces algorithm, the recognition rate rose from 82.8% to 84.7% due to the more accurate generation of motion-free texture maps.

Conclusion and Future Work

By fusing raw pixel-level gradients rather than final tracking results, this method effectively addresses the nonfrontal pose problem. While the current implementation uses a generic cylindrical model and requires manual initialization, future work aims to automate initialization and extend the fusion technique to more complex 3D meshes or different object classes like vehicles. Additionally, the authors suggest exploring distributed computing to manage the increased computational load of joint multi-camera optimization.

Summary updated

BibTeX
@inproceedings{harguess2011full,
  author = {Josh Harguess and Changbo Hu and J. K. Aggarwal},
  title = {{Full-motion recovery from multiple video cameras applied to face tracking and recognition}},
  booktitle = {Workshops @ ICCV},
  year = {2011},
  doi = {10.1109/iccvw.2011.6130479},
}
Figure