AAM and Non-rigid Registration in Augmented Reality

Yuan Tian, Tao Guan, Cheng WangInTech 2010 · DOI · OpenAlex

Abstract

Augmented Reality (AR) requires precise alignment between virtual graphics and real-world objects. While registration for rigid objects is well-established, tracking non-rigid, deforming surfaces like human hands remains a significant challenge. This paper proposes a robust framework that integrates Active Appearance Models (AAM) for 2D feature tracking with a factorization algorithm for 3D shape and camera pose reconstruction. By reducing complex deformations into a low-dimensional set of parameters, the system achieves real-time, stable registration of virtual objects onto moving and flexing surfaces.

The Challenge of Non-Rigid Registration

In AR, "registration" refers to the spatial alignment of virtual data with the physical environment. For static or rigid objects, such as tables or posters, this is typically handled by tracking a fixed set of features. However, the real world is dynamic; human faces change expressions, and hands flex and rotate. Tracking these non-rigid objects requires estimating not only the camera's position but also a high number of parameters describing the object's internal deformation. Historically, the computational cost of this dual estimation has been a barrier to real-time AR.

Methodology

The researchers address this by splitting the process into two phases: offline modeling and online tracking.

Active Appearance Models (AAM)

The core of the 2D tracking system is the Active Appearance Model. This statistical model captures variations in both shape (landmark coordinates) and texture (pixel intensities). By applying Principal Component Analysis (PCA), the system reduces the high-dimensional data of a deforming hand into a few "configuration weights."

Flowchart of the AAM construction and fitting process.

3D Reconstruction and Optimization

To move from 2D tracking to 3D registration, the authors employ a factorization algorithm. This method decomposes a sequence of 2D tracking data into a 3D shape basis and the camera's rotation. During live tracking, a Levenberg-Marquardt optimization loop fine-tunes these parameters, ensuring that virtual objects are "pinned" to the deforming surface with minimal projection error.

Datasets and Implementation

The study utilized custom-captured sequences to validate the approach. For the hand model, the authors manually labeled 36 landmarks across a set of training images to establish the initial AAM.

Manual landmarking of a human hand for AAM training.

The system was also tested on face tracking sequences to demonstrate its versatility across different non-rigid targets. The implementation was carried out in C using standard libraries like OpenGL and OpenCV, achieving real-time performance on a 1.7GHz processor.

Key Insights and Results

  • Dimensionality Reduction: Using PCA within the AAM framework makes tracking complex deformations computationally feasible.
  • Metric Constraints: Enforcing orthonormality in rotation matrices during factorization is crucial for recovering accurate 3D geometry from 2D images.
  • Robust Augmentation: The system successfully demonstrated a virtual teapot remaining realistically attached to a moving, flexing hand, proving robust against lighting changes and viewpoint shifts.

Limitations and Future Directions

While effective, the current pipeline requires significant manual effort for initial landmarking. Future work aims to automate this process. Additionally, while the model handles minor deformations well, extreme self-occlusions (e.g., a finger disappearing behind the palm) remain a challenge. Scaling the system to work across different users without individual training is also a key goal for broader AR applications.

Summary updated

Abstract

Este volumen se ha propuesto incluir no solamente los debates teóricos y metodológicos con respecto a la posible definición del microrrelato como "cuarto género" y los diversos análisis sobre el desarrollo y la historia del mismo. Más bien, relacionamos la minificción literaria con otras prácticas simbólicas, y consideramos las nuevas posibilidades de difusión de la minificción en los medios masivos de comunicación y, sobre todo, en las redes sociales (Facebook, Twitter) y en el Internet en general. En los cuatro apartados del volumen, sus autores se ocupan de la teoría del género y la historia del micorrelato literario; analizan la intertextualidad del nuevo género; interpretan una serie de minificciones literarias de autoras y autores hispanoamericanos y españoles; y consideran otras formas de lo micromediático, los litblogs, la producción de microrrelatos en las redes sociales, y las minificciones cinematográficas. OTTMAR ETTE es catedrático de Literaturas Románicas en la Universidad de Potsdam. DIETER INGENSCHAY, catedrático de Literaturas Románicas en la Universidad Humboldt de Berlín. FRIEDHELM SCHMIDT-WELLE es investigador en Literatura y Estudios Culturales en el Instituto Ibero-Americano de Berlín y FERNANDO VALLS, catedrático de Filología Española en la Universidad Autónoma de Barcelona.
BibTeX
@inproceedings{tian2010aam,
  author = {Yuan Tian and Tao Guan and Cheng Wang},
  title = {{AAM and Non-rigid Registration in Augmented Reality}},
  booktitle = {InTech},
  year = {2010},
  doi = {10.5772/7132},
}
Figure