ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis

Christian Vogler, Dimitris N. MetaxasICCV 1998 · OpenReview

TL;DR

This paper introduces a novel framework for recognizing American Sign Language (ASL) by combining 3D motion analysis with Hidden Markov Models (HMMs). They demonstrate that using 3D features significantly improves recognition accuracy over 2D, and that coupling computer vision with HMMs offers a promising path to overcome limitations in current ASL recognition systems.

What Problem Does This Paper Address?

Recognizing American Sign Language (ASL) is a challenging task due to its complex grammatical structure, the inherent variability in sign execution, and the difficulty in segmenting continuous sign streams. Previous approaches often relied on 2D image data, which can lose crucial spatial information, or used input devices like data gloves, which are less natural than vision-based methods. This paper aims to improve ASL recognition accuracy, particularly for continuous signing, by leveraging richer 3D motion data and a more sophisticated integration of computer vision and HMMs.

Methods

The core of the proposed framework involves two main components:

  • 3D Motion Tracking: They employ a physics-based, multi-view computer vision system to track the 3D motion of a signer's arms and hands. This system uses deformable models and geometric principles to estimate the 3D position and orientation of body parts from multiple camera views, even in the presence of occlusions. For ASL recognition, they focus on arm motion, as it carries significant information.
  • Hidden Markov Models (HMMs): HMMs are used for the actual recognition of isolated and continuous ASL sentences. The paper discusses standard HMM theory, including definitions, the three fundamental problems (probability computation, state sequence recovery, and parameter training), and the Viterbi algorithm for continuous recognition.
  • Coupling Vision and HMMs: A novel aspect is the temporal segmentation of the 3D data stream using vision methods. This segmentation identifies geometric primitives (lines, planes, holds) within the motion. These geometric properties are then used to constrain the HMM framework, effectively providing a "backup check" for the HMM recognition hypotheses. This coupling aims to leverage the strengths of both approaches: HMMs for automatic segmentation and temporal modeling, and computer vision for robust feature extraction and dynamic weighting.
  • Context-Dependent HMMs: To address coarticulation (how signs influence each other), they explore context-dependent HMMs, specifically bi-sign models, where the recognition of a sign is influenced by the preceding sign.

Datasets

  • Vocabulary: A 53-sign vocabulary was used, chosen to represent natural conversation and make extensive use of signing space.
  • Data Collection:
    • 486 continuous ASL sentences were collected, each 2-12 signs long, totaling 2345 signs.
  • For isolated recognition, 656 examples of 40 signs were collected, with at least 6 examples per sign for training and 2 for testing.
  • Data was collected using both their custom computer vision system and an Ascension Technologies Flock of Birds system (for prototyping due to speed).

Key Insights and Ideas

  • 3D is Better than 2D: Three-dimensional features derived from motion tracking significantly outperform 2D features for ASL recognition, indicating that the loss of depth information in 2D is detrimental.
  • Vision as a Constraint: Coupling the computer vision-based motion analysis with HMMs provides a powerful mechanism to constrain the recognition process. The geometric primitives extracted by vision can validate or reject HMM-generated sign sequences, improving robustness.
  • Context Matters: Context-dependent HMMs (specifically bi-sign models) show an improvement in continuous ASL recognition, highlighting the importance of modeling how signs influence each other.
  • Feature Selection is Crucial: The performance of HMMs is highly sensitive to the choice of features. Position coordinates proved more robust than velocity features, likely because HMMs can naturally handle variations in sign duration by staying in states longer.
  • Limitations of Pure HMMs: HMMs alone struggle with insufficient training data and have limitations in dynamically weighting features or handling violations of stochastic independence assumptions. The proposed coupling aims to mitigate these.

Key Results

  • Isolated Recognition: 3D features achieved accuracies around 98%, outperforming 2D features (around 97.75%). Polar coordinates showed a slight advantage over Cartesian.
  • Continuous Recognition:
    • 3D context-independent HMMs achieved 87.71% word accuracy.
  • 3D context-dependent HMMs improved this to 89.91%.
  • 2D context-dependent HMMs performed worse at 83.63%.
  • Coupling Experiments: The coupled vision-HMM system showed promising results, with the motion analysis rejecting 10 out of 97 sentences. In one experiment, the coupling achieved 90.10% word accuracy, slightly better than context-dependent HMMs alone.

Limitations and Future Work

  • Limitations:
    • Training Data: The performance of context-dependent models is significantly hampered by insufficient training data.
  • Segmentation: The vision-based segmentation process can be prone to spurious or missed segments.
  • Coarticulation Modeling: While context-dependent HMMs offer some improvement, the paper notes that movement epenthesis (a specific coarticulation phenomenon) might be better modeled explicitly rather than through context-dependent HMMs alone.
  • Grammar Complexity: Using a strong, explicit grammar for continuous recognition is impractical due to exponential network growth.
  • Hand Configuration: The current work primarily uses arm motion, neglecting hand configuration, which is crucial for distinguishing some signs.
  • Future Work:
    • Larger Corpora: Developing a standardized corpus of real-world ASL conversations is essential for robust statistical language modeling and comparison across research.
  • Larger Vocabularies: Testing algorithms with larger vocabularies to assess scalability.
  • Facial Expressions and Hand Configuration: Incorporating facial expressions and hand configuration information into the recognition framework.
  • More Robust Vision Analysis: Improving the robustness of the vision system, fine-tuning sign templates, and dynamically weighting features.
  • Anthropometrically Correct Models: Developing more accurate hand models for better vision tracking.
  • Explicit Coarticulation Modeling: Further research into explicitly modeling movement epenthesis.

Summary updated

Abstract

We present a framework for recognizing isolated and continuous American Sign Language (ASL) sentences from three-dimensional data. The data are obtained by using physics-based three-dimensional tracking methods and then presented as input to Hidden Markov Models (HMMs) for recognition. To improve recognition performance, we model context-dependent HMMs and present a novel method of coupling three-dimensional computer vision methods and HMMs by temporally segmenting the data stream with vision methods. We then use the geometric properties of the segments to constrain the HMM framework for recognition. We show in experiments with a 53 sign vocabulary that three-dimensional features outperform two-dimensional features in recognition performance. Furthermore, we demonstrate that context-dependent modeling and the coupling of vision methods and HMMs improve the accuracy of continuous ASL recognition.
BibTeX
@inproceedings{vogler1998asl,
  author = {Christian Vogler and Dimitris N. Metaxas},
  title = {{ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1998},
  url = {https://openreview.net/forum?id=r1WYH-fdWr},
}
Figure