Abstract
Recognition of gesture sequences is in general a very difficult problem, but in certain domains the difficulty may be mitigated by exploiting the domain's "grammar". One such grammatically constrained gesture sequence domain is sign language. In this paper we investigate the case of finger spelling recognition, which can be very challenging due to the quick, small motions of the fingers. Most prior work on this task has assumed a closed vocabulary of finger spelled words, here we study the more natural open-vocabulary case, where the only domain knowledge is the possible finger spelled letters and statistics of their sequences. We develop a semi-Markov conditional model approach, where feature functions are defined over segments of video and their corresponding letter labels. We use classifiers of letters and linguistic hand shape features, along with expected motion profiles, to define segmental feature functions. This approach improves letter error rate (Levenshtein distance between hypothesized and correct letter sequences) from 16.3% using a hidden Markov model baseline to 11.6% using the proposed semi-Markov model.
Introduction
Fingerspelling is a vital component of American Sign Language (ASL), used for names, technical terms, and "loan words" from English. However, it is notoriously difficult for computer vision systems to track. Unlike the broad, sweeping motions of general signs, fingerspelling involves small, rapid movements of the fingers that often blur together in standard video.
Most prior research simplified the problem by using a "closed vocabulary"—essentially a multiple-choice test where the system only had to choose between 20 and 100 known words. This paper addresses the unconstrained (open-vocabulary) case, which is much more realistic but significantly harder because the system must recognize any sequence of the 26 letters based solely on visual cues and the statistical likelihood of letter transitions (such as "q" usually being followed by "u").
Methodology: Semi-Markov Conditional Random Fields
The core innovation of this work is the use of Semi-Markov Conditional Random Fields (SCRFs). In a standard Hidden Markov Model (HMM) or a typical CRF, the system tries to assign a label to every single frame of video. However, a single letter spans many frames, and the "essence" of that letter is often found in the movement across those frames, not in a single snapshot.
Key components of the SCRF approach:
- Segmental Modeling: Instead of labeling individual frames, the SCRF labels segments of video. This allows the model to calculate "feature functions" over the entire duration of a letter, such as the average handshape or the total motion profile.
- Linguistic Features: Rather than just looking at raw pixels, the authors trained neural networks to recognize "phonological" features—linguistic building blocks like which fingers are "active," whether the thumb is opposed, or if the fingers are flexed.
- Peak Detection: Fingerspelling has a rhythm. There is usually a "peak" of articulation where the hand reaches the target shape and motion briefly slows down. The authors built a specific feature to detect these local minima in motion to help the model find letter boundaries.
- Two-Stage Inference: Because checking every possible segment start and end time is computationally expensive, the researchers used a standard HMM to generate a "shortlist" of candidate segmentations, which the SCRF then re-ranked to find the best fit.
Experimental Setup
The study utilized video recordings of four native ASL signers. Each signer performed a list of 300 words (including names and common nouns), each signed twice, for a total of 600 word instances per signer. The video was captured at 60 frames per second in a controlled studio environment. The data was manually annotated by multiple experts to identify the "peaks" of each letter, providing a high-quality ground truth for training.
Results and Discussion
The researchers compared their SCRF model against two baselines: a standard HMM and a "Tandem HMM" (which uses neural network outputs as observations).
| Model | Letter Error Rate (LER) |
|---|---|
| Baseline HMM | 16.3% |
| Tandem HMM | 14.5% |
| Proposed SCRF | 11.6% |
This represents a 29% relative reduction in error compared to the standard HMM. The SCRF was particularly effective at correcting mistakes where the baseline model would skip letters or misidentify the boundaries between them.
Key Insights
- Segments > Frames: Gestures are inherently temporal. By defining features over a segment, the model captures the dynamics of signing much better than a frame-by-frame classifier ever could.
- Linguistic Abstraction: Training classifiers for specific handshape properties (like "is the index finger extended?") makes the model more robust than mapping raw descriptors directly to a letter.
- The "Peak" Matters: Recognizing that motion slows down at the moment a letter is most clearly formed allowed the authors to create a "peak detection" feature that acts as a natural anchor for the segmentation process.
Limitations and Future Work
While a major step forward, the paper notes several hurdles. Moving to a "signer-independent" system—where the model can recognize someone it has never seen before—remains a significant challenge due to variations in hand anatomy and signing style. Furthermore, the SCRF is computationally slower than an HMM, necessitating the two-stage approach. Finally, hand localization still required some manual calibration, though the authors suggest this could be automated in the future.
Ultimately, this work demonstrated that for complex, fluid gestures like fingerspelling, looking at the "big picture" of a segment is far more effective than analyzing individual frames.
Summary updated