Abstract
Augmented Reality (AR) requires precise alignment between virtual graphics and real-world objects. While registration for rigid objects is well-established, tracking non-rigid, deforming surfaces like human hands remains a significant challenge. This paper proposes a robust framework that integrates Active Appearance Models (AAM) for 2D feature tracking with a factorization algorithm for 3D shape and camera pose reconstruction. By reducing complex deformations into a low-dimensional set of parameters, the system achieves real-time, stable registration of virtual objects onto moving and flexing surfaces.
The Challenge of Non-Rigid Registration
In AR, "registration" refers to the spatial alignment of virtual data with the physical environment. For static or rigid objects, such as tables or posters, this is typically handled by tracking a fixed set of features. However, the real world is dynamic; human faces change expressions, and hands flex and rotate. Tracking these non-rigid objects requires estimating not only the camera's position but also a high number of parameters describing the object's internal deformation. Historically, the computational cost of this dual estimation has been a barrier to real-time AR.
Methodology
The researchers address this by splitting the process into two phases: offline modeling and online tracking.
Active Appearance Models (AAM)
The core of the 2D tracking system is the Active Appearance Model. This statistical model captures variations in both shape (landmark coordinates) and texture (pixel intensities). By applying Principal Component Analysis (PCA), the system reduces the high-dimensional data of a deforming hand into a few "configuration weights."
3D Reconstruction and Optimization
To move from 2D tracking to 3D registration, the authors employ a factorization algorithm. This method decomposes a sequence of 2D tracking data into a 3D shape basis and the camera's rotation. During live tracking, a Levenberg-Marquardt optimization loop fine-tunes these parameters, ensuring that virtual objects are "pinned" to the deforming surface with minimal projection error.
Datasets and Implementation
The study utilized custom-captured sequences to validate the approach. For the hand model, the authors manually labeled 36 landmarks across a set of training images to establish the initial AAM.
The system was also tested on face tracking sequences to demonstrate its versatility across different non-rigid targets. The implementation was carried out in C using standard libraries like OpenGL and OpenCV, achieving real-time performance on a 1.7GHz processor.
Key Insights and Results
- Dimensionality Reduction: Using PCA within the AAM framework makes tracking complex deformations computationally feasible.
- Metric Constraints: Enforcing orthonormality in rotation matrices during factorization is crucial for recovering accurate 3D geometry from 2D images.
- Robust Augmentation: The system successfully demonstrated a virtual teapot remaining realistically attached to a moving, flexing hand, proving robust against lighting changes and viewpoint shifts.
Limitations and Future Directions
While effective, the current pipeline requires significant manual effort for initial landmarking. Future work aims to automate this process. Additionally, while the model handles minor deformations well, extreme self-occlusions (e.g., a finger disappearing behind the palm) remain a challenge. Scaling the system to work across different users without individual training is also a key goal for broader AR applications.
Summary updated