A Hybrid Grammar-Based Approach for Learning and Recognizing Natural Hand Gestures

Amir Sadeghipour, Stefan KoppAAAI 2014 · DOI

Introduction

Recognizing natural "iconic" gestures—those used spontaneously in conversation to describe shapes or movements—presents a significant challenge for computer vision. Unlike rigid command gestures, iconic gestures exhibit high variability in execution across different individuals. Traditional approaches typically rely on either statistical models (e.g., HMMs), which handle noisy data well but struggle with hierarchical structures, or grammar-based models, which capture structure but are often too brittle for real-world data.

Methodology: Feature-based Stochastic Context-Free Grammar (FSCFG)

To address these limitations, this research introduces the Feature-based Stochastic Context-Free Grammar (FSCFG). This hybrid framework integrates high-level structural rules with low-level statistical features.

The FSCFG architecture bridges syntactic grammar rules with statistical feature sets.

In this model, terminal symbols are defined as probabilistic clusters of features (multidimensional Gaussians) rather than static symbols. This allows the grammar to operate directly on raw trajectory data. The system learns the grammar structure through Bayesian operators like "Merge" (combining similar symbols) and "Chunk" (identifying common sequences). To ensure robustness, the model incorporates skip symbols and epsilon transitions to handle noise and missing data segments.

The 3D Iconic Gestures (3DIG) Dataset

The study utilized the 3D Iconic Gestures (3DIG) dataset, comprising 1,739 gestures from 29 participants describing 20 different 3D objects. Data was captured using a Microsoft Kinect, providing 3D skeletal tracking.

Distribution of gesture strategies (drawing, posturing, enacting) across different objects in the 3DIG dataset.

Analysis of the dataset revealed that "drawing" the contours of an object is the most prevalent strategy. Consequently, wrist trajectories serve as a highly effective feature for recognition. The model automatically distinguishes between invariant features (essential to the gesture's meaning) and variant features (noise such as speed or hand choice).

Results and Performance

The FSCFG model consistently outperformed baseline Hidden Markov Models (HMMs) and Support Vector Machines (SVMs). Notably, for "drawing" gestures, the model achieved recognition rates that surpassed human observers viewing the same skeletal motion data. In specific subsets, such as synchronized drawing gestures, the model reached an Area Under the Curve (AUC) of 0.996.

Conclusion and Future Work

While highly effective for trajectory-based gestures, the current wrist-centric approach struggles with "static posturing" and "enacting" techniques where hand shape is paramount. Future research aims to incorporate finger-level tracking and hand-shape features into the FSCFG framework to enhance its versatility across all natural gesture types.

Summary updated

BibTeX
@inproceedings{sadeghipour2014hybrid,
  author = {Amir Sadeghipour and Stefan Kopp},
  title = {{A Hybrid Grammar-Based Approach for Learning and Recognizing Natural Hand Gestures}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.9023},
}

Loading related content...

Figure