Abstract
Deriving human body shape and motion from optical or magnetic motion-capture data is an inherently difficult task. The body is very complex and the data is rarely error-free and often incomplete. The task becomes even more difficult if one attempts to use video-data instead, because it is much noisier. In the last few years, we have developed techniques that use sophisticated human animation models to fit such noisy and incomplete data. It is acquired using a variety of devices, ranging from sophisticated optical motion-capture systems to ordinary video-cameras. We use facial and body animation models, not only to represent the data, but also to guide the fitting process, thereby substantially improving performance.
Introduction
Recovering accurate human shape and motion from sensor data is a cornerstone of computer graphics and biomechanics. However, traditional methods often struggle with noisy or incomplete data, such as marker occlusions in motion capture (MoCap) or low-resolution video. This paper presents a robust framework that leverages sophisticated animation models as "priors" to bridge the gap between raw sensor data and realistic digital humans. By using layered models—incorporating skeletons, muscle-simulating metaballs, and skin—the system can intelligently interpret ambiguous data.
Model-Driven Fitting Methods
The core of the approach is using the animation model not just as an output, but as a guide during the fitting process. This "knowledge-based" approach allows the system to intelligently handle missing or noisy data points by enforcing biological constraints.
Layered Body and Facial Models
The researchers utilize a multi-layered representation to simulate human anatomy. For the body, this includes a hierarchical skeleton, "metaballs" (ellipsoidal primitives) to simulate the volume of muscle and fat, and a polygonal "skin" mesh. For the head, a mesh controlled by Rational Free Form Deformations (RFFDs) is used to simulate muscle activity and expressions.
This anatomical knowledge allows the system to maintain tracking even when markers are occluded, as the model's physical constraints and volume help predict missing joint positions. In video-based tracking, a generic face model regularizes the bundle adjustment process, allowing for motion and shape estimation from ordinary video with minimal manual input—requiring only five key feature points to be identified in a single frame.
Experimental Setup and Datasets
The framework was evaluated using diverse datasets, including complex "Karate" and "Gym" motions captured with an eight-camera infrared system. For facial modeling, high-precision 3D models from a Minolta laser scanner served as ground truth to benchmark the video-based reconstruction.
Key Results and Insights
The integration of animation models significantly enhances robustness. In the "Karate" MoCap sequence, which features high-speed movements and frequent occlusions, the model-based approach maintained 100% marker identification, whereas standard tracking methods failed within 200 frames. For facial modeling, the system achieved a median accuracy of approximately 1mm compared to laser-scanned ground truth, demonstrating the power of model-based regularization even with uncalibrated video. The system also successfully recovered complex "naked" upper-body motions involving severe self-occlusion.
Conclusion
By treating the animation model as an active participant in data fitting, this research demonstrates a path toward more automated and robust human motion recovery. Future work aims to capture real-time facial dynamics in conjunction with body movement to create comprehensive biometric models of individual movement styles.
Summary updated