Introduction
Recognizing natural "iconic" gestures—those used spontaneously in conversation to describe shapes or movements—presents a significant challenge for computer vision. Unlike rigid command gestures, iconic gestures exhibit high variability in execution across different individuals. Traditional approaches typically rely on either statistical models (e.g., HMMs), which handle noisy data well but struggle with hierarchical structures, or grammar-based models, which capture structure but are often too brittle for real-world data.
Methodology: Feature-based Stochastic Context-Free Grammar (FSCFG)
To address these limitations, this research introduces the Feature-based Stochastic Context-Free Grammar (FSCFG). This hybrid framework integrates high-level structural rules with low-level statistical features.
In this model, terminal symbols are defined as probabilistic clusters of features (multidimensional Gaussians) rather than static symbols. This allows the grammar to operate directly on raw trajectory data. The system learns the grammar structure through Bayesian operators like "Merge" (combining similar symbols) and "Chunk" (identifying common sequences). To ensure robustness, the model incorporates skip symbols and epsilon transitions to handle noise and missing data segments.
The 3D Iconic Gestures (3DIG) Dataset
The study utilized the 3D Iconic Gestures (3DIG) dataset, comprising 1,739 gestures from 29 participants describing 20 different 3D objects. Data was captured using a Microsoft Kinect, providing 3D skeletal tracking.
Analysis of the dataset revealed that "drawing" the contours of an object is the most prevalent strategy. Consequently, wrist trajectories serve as a highly effective feature for recognition. The model automatically distinguishes between invariant features (essential to the gesture's meaning) and variant features (noise such as speed or hand choice).
Results and Performance
The FSCFG model consistently outperformed baseline Hidden Markov Models (HMMs) and Support Vector Machines (SVMs). Notably, for "drawing" gestures, the model achieved recognition rates that surpassed human observers viewing the same skeletal motion data. In specific subsets, such as synchronized drawing gestures, the model reached an Area Under the Curve (AUC) of 0.996.
Conclusion and Future Work
While highly effective for trajectory-based gestures, the current wrist-centric approach struggles with "static posturing" and "enacting" techniques where hand shape is paramount. Future research aims to incorporate finger-level tracking and hand-shape features into the FSCFG framework to enhance its versatility across all natural gesture types.
Summary updated