TL;DR
This paper presents a method to automatically reconstruct realistic 3D human body models and their motion from multi-camera video sequences. By combining robust 3D tracking of key points and surface features with a novel fitting approach to a simplified metaball-based body model, the authors demonstrate the ability to capture both shape and movement with minimal user intervention, paving the way for more accessible and efficient human animation.
What Problem Does This Paper Address?
Creating realistic 3D human animation models is a time-consuming and labor-intensive process, often requiring skilled designers. While existing methods can capture surface details, they struggle with dynamic motion and require significant manual effort for both model creation and animation. The authors aim to automate this process, enabling the creation of personalized, realistic human animation models from simple video recordings, making sophisticated animation accessible to a broader audience.
Methods
The core of the approach involves two main stages: 3D data extraction and model fitting.
- 3D Data Extraction:
- Image Acquisition: Synchronized video from three CCD cameras arranged linearly is used. To handle the "sawtooth" pattern caused by interlaced scanning of moving objects, only odd lines of the images are processed, reducing vertical resolution by 50%.
- System Calibration: A reference bar with retroreflective targets is used for self-calibrating bundle adjustment to determine camera parameters and orientation.
- Surface Measurement: An adaptive least squares matching (LSM) stereo matcher is employed. It uses an image patch as a template and searches for its corresponding match in other images, allowing for affine transformations and gray-level variations. This process is initialized with manually selected seed points.
- 3D Tracking: A novel tracking process based on LSM is introduced. It tracks both spatial correspondences between different camera views and temporal correspondences between subsequent frames. For temporal tracking, a linear prediction of the point's position is made, and then LSM is applied within a search box around this prediction. This is done independently for each camera view, followed by a spatial LSM to refine the 3D position. Velocities and accelerations are also computed.
- Surface Tracking: The LSM tracking algorithm is applied to all points measured on the surface at the beginning of the sequence, resulting in a vector field of trajectories. This field is then filtered using thresholds for velocity and acceleration, and a local uniformity check to ensure consistency in movement.
- Model Fitting:
- Simplified Model: A simplified, layered human body model is used for fitting. This model consists of a skeleton and metaballs (ellipsoids) attached to each limb to represent gross body shape (muscles, fat). This is a simplification of a more complex, complete animation model.
- Least Squares Framework: The fitting process minimizes an objective function using a standard least-squares framework. Observation equations are formulated based on image data (stereo points) and the model's predicted positions.
- Metaball Potential Function: An exponential field function is used for metaballs, which is computationally efficient for derivatives needed in fitting and ensures smooth blending. The distance function is based on the I/O distance of an ellipsoid.
- Iterative Optimization: The fitting is performed iteratively, alternating between optimizing primitive (metaball radii) parameters and skeleton (joint angles) parameters to achieve better convergence.
Datasets
The paper uses video sequences of a person's arm wearing a textured sweater. The authors mention that their final goal is to handle a wider range of textures, including bare skin. No specific public datasets are mentioned or introduced.
Key Insights and Ideas
- Automating Human Animation: The overarching goal of automating the creation of realistic human animation models from video is a significant step towards making sophisticated animation more accessible and cost-effective.
- Leveraging LSM for Tracking: The paper demonstrates the effectiveness of using least squares matching not only for stereo correspondence but also for robust temporal tracking of points across video frames, even with textured surfaces.
- Simplified Model for Fitting: The use of a simplified metaball-based model, despite its limitations in capturing fine details, is crucial for enabling robust fitting to noisy 3D data derived from video. This highlights a practical trade-off between model complexity and fitting accuracy.
- Iterative Fitting Strategy: Alternating between fitting skeleton parameters and metaball parameters is a key insight for improving convergence in the non-linear optimization process.
- Surface Consistency Checks: Filtering the surface tracking results for velocity, acceleration, and local uniformity is essential for removing erroneous measurements and ensuring the integrity of the reconstructed surface.
Key Results
- The authors successfully demonstrate the ability to track six single points through a sequence and reconstruct their 3D trajectories (Figure 8).
- They show the tracking of the surface, visualized as a vector field of trajectories (Figure 9).
- The fitting of their simplified arm model to image data is presented, showing reconstructed positions and shapes (Figure 13).
- The recovered animation parameters are applied to a completely different virtual human model, demonstrating the transferability of the motion data (Figure 14).
- The paper shows that the recovered shape and motion parameters can be used to reconstruct the original movement or to drive other animation models.
Limitations and Future Work
- Simplified Model: The simplified model used for fitting only allows for a good approximation of the skeleton but a rather rough approximation of the shape. The authors plan to refine the model to the full set of primitives for more realistic shape modeling.
- Texture Handling: While the current approach works with textured sweaters, the ultimate goal is to handle a wider range of textures, including bare skin.
- Occlusion and Model Guidance: The authors plan to investigate allowing the model to guide the tracking process, particularly to predict the reappearance of occluded points.
- Synthetic Data Testing: Future work includes generating synthetic data to test the system's accuracy.
- Full Body Motion: The current application is shown on an arm, but the algorithms are being applied to full body motion sequences.
- Parameter Tuning: The paper mentions that the weight parameter in the metaball function might be left as a fitting parameter in the future to allow for sharper edges.
Summary updated