Spelling it out: Real-time ASL fingerspelling recognition

Nicolas Pugeault, Richard BowdenWorkshops @ ICCV 2011 · OpenReview

Abstract

This article presents an interactive hand shape recognition user interface for American Sign Language (ASL) finger-spelling. The system makes use of a Microsoft Kinect device to collect appearance and depth images, and of the OpenNI+NITE framework for hand detection and tracking. Hand-shapes corresponding to letters of the alphabet are characterized using appearance and depth images and classified using random forests. We compare classification using appearance and depth images, and show a combination of both lead to best results, and validate on a dataset of four different users. This hand shape detection works in real-time and is integrated in an interactive user interface allowing the signer to select between ambiguous detections and integrated with an English dictionary for efficient writing.

Introduction

While sign language recognition has progressed, "fingerspelling"—the practice of spelling out words letter-by-letter—remains a significant hurdle for computer vision. Many ASL letters are visually nearly identical; for instance, the letters a, e, m, n, s, and t are all represented by a closed fist, differing only by the subtle placement of the thumb.

Beyond visual similarity, traditional camera-based systems struggle with varying lighting conditions and different skin tones. Furthermore, most systems are "black boxes" that either guess correctly or fail entirely, offering no intuitive way for a user to correct a misclassification in real-time. Researchers at the University of Surrey have developed a robust, multi-modal sensing approach and an interactive user interface to address these challenges.

Methodology

The system is built around the Microsoft Kinect, which provides both standard intensity (color) images and depth maps. The processing pipeline consists of several key stages:

Overview of the ASL fingerspelling recognition system pipeline.
  1. Hand Extraction: Using the OpenNI+NITE framework, the system tracks the user's hand in 3D space. It segments the hand from the background by assuming the hand is a continuous region within a 20cm depth range.
  2. Feature Extraction: To describe the hand shapes, the team used a bank of Gabor filters (at 4 scales and 4 orientations). These filters capture both the broad silhouette of the hand and the fine details of finger positioning. These responses are averaged into a 1,024-dimensional feature vector for both depth and intensity.
  3. Classification: A Multi-class Random Forest algorithm is employed for classification. This algorithm handles high-dimensional data efficiently and runs at 30Hz, enabling real-time performance.
  4. Interactive UI: When the classifier is uncertain, the UI displays a semi-circle of the most likely letter candidates above the user's hand. The user can select the correct letter with a slight hand movement. The system also integrates the Aspell dictionary to suggest word completions.

Datasets

The researchers collected a substantial new dataset for this project, comprising 48,000 samples from 4 different individuals. Users were asked to move their hands while keeping a fixed sign shape to capture different angles, distances, and backgrounds. The dataset covers 24 of the 26 ASL letters, excluding 'j' and 'z' as they require motion.

Key Insights and Results

The study yielded several important findings:

  • Multi-modal Superiority: Depth information is more robust to environmental changes like lighting than color. While intensity alone reached 73% precision and depth alone reached 69%, the combined multi-modal approach achieved a mean precision of 75%.
  • Collaborative Design: Rather than aiming for 100% autonomous accuracy on visually similar "fist letters," the system uses a human-in-the-loop approach. By presenting the top hypotheses, it turns potential failures into seamless interactions.
  • Temporal Filtering: To avoid jitter during transitions, the system only attempts recognition when the hand's velocity and depth variance fall below a certain threshold.

As expected, the letter t was the most difficult (37% precision) due to its similarity to a, e, n, and s. Letters with distinct finger extensions, like l and v, performed significantly better.

Conclusion and Future Work

The current system successfully demonstrates real-time static shape recognition. Future work will focus on incorporating language modeling (using priors for letter sequences), user adaptation to improve individual accuracy, and expanding the feature set to include motion descriptors for dynamic signs like 'j' and 'z'.

Summary updated

Abstract

This article presents an interactive hand shape recognition user interface for American Sign Language (ASL) finger-spelling. The system makes use of a Microsoft Kinect device to collect appearance and depth images, and of the OpenNI+NITE framework for hand detection and tracking. Hand-shapes corresponding to letters of the alphabet are characterized using appearance and depth images and classified using random forests. We compare classification using appearance and depth images, and show a combination of both lead to best results, and validate on a dataset of four different users. This hand shape detection works in real-time and is integrated in an interactive user interface allowing the signer to select between ambiguous detections and integrated with an English dictionary for efficient writing.
BibTeX
@inproceedings{pugeault2011spelling,
  author = {Nicolas Pugeault and Richard Bowden},
  title = {{Spelling it out: Real-time ASL fingerspelling recognition}},
  booktitle = {Workshops @ ICCV},
  year = {2011},
  url = {https://openreview.net/forum?id=SInxR6pM5xW},
}
Figure