Appearance Based Recognition Methodology for Recognising Fingerspelling Alphabets

M. G. Suraj, D. S. GuruIJCAI 2007

Abstract

Automatic sign language recognition is a vital tool for social integration, yet it remains a significant computer vision challenge due to environmental variability and the complexity of hand gestures. This paper demonstrates that the performance of Principal Component Analysis (PCA) in recognizing fingerspelling alphabets can be significantly enhanced through targeted image pre-processing. By applying techniques such as adaptive smoothing, edge enhancement, and gray-level reduction, the authors achieve high recognition rates on a diverse real-world dataset without the need for expensive hardware like cyber gloves.

Introduction

Historically, sign language recognition has relied on two main approaches: hardware-heavy methods using "cyber gloves" or accelerometers, and vision-based methods using standard cameras. While hardware methods are accurate, they are expensive and cumbersome for everyday use. Vision-based methods are more accessible but often require highly controlled environments to work reliably. This research addresses the limitations of Appearance-Based Recognition, specifically PCA, which is computationally efficient but sensitive to noise and variability in lighting or hand structure. The goal is to make PCA robust enough to handle a large, varied dataset of 20 different individuals in a real-world setting.

Methodology

The authors utilize a standard PCA (Eigenfaces) pipeline, introducing a critical pre-processing stage before training and recognition. Four distinct enhancement techniques were evaluated:

  1. Wiener Filtering: An adaptive smoothing process that reduces noise while preserving important edges by adjusting to local image variance.
  2. Extended Sobel Filter: A novel application where a horizontal Sobel mask detects edges, which are then added back to the original image ($K = I + E$) to create a high-contrast, edge-emphasized version of the sign.
  3. Histogram Equalization: A technique to increase local contrast by re-distributing the gray values of the pixels.
  4. Gray Level Reduction: The image is reduced from 256 gray levels down to just 3 (quantized to specific values like 50, 150, and 200). This "flattens" the image to highlight the most prominent geometric features.
Example of a captured fingerspelling alphabet sign used in the study.

For classification, the standard Euclidean distance measure in the reduced Eigen-space is employed.

Experimental Setup and Dataset

A custom, robust dataset was constructed to test the methodology:

  • Size: 2,880 total images.
  • Content: 24 static fingerspelling alphabets (excluding dynamic signs 'j' and 'z').
  • Diversity: Images were captured from 20 different individuals using a standard webcam against a plain background.
  • Split: 4 samples per alphabet per user for training (1,920 images) and 2 samples for testing (960 images).

Key Results and Insights

The experimental results demonstrated a clear advantage for the proposed pre-processing techniques. Baseline PCA achieved a maximum recognition rate of 84.3%, while PCA combined with the Extended Sobel filter reached 86.4%. When combined with Diagonal PCA, the accuracy peaked at 90.17%.

Processed image demonstrating the effect of enhancement techniques like edge detection or gray-level reduction.

Analysis

  • Edges are Essential: The success of the Extended Sobel filter suggests that the outline and internal boundaries of the hand are more informative than skin texture or lighting gradients.
  • Information Bottlenecks: The Gray Level Reduction technique performed surprisingly well, achieving 86.7% accuracy with only 30 dimensions. By stripping away 98% of the color depth, the algorithm focuses on core geometry, reducing noise from skin tones or lighting shifts.
  • Universal Boosting: These pre-processing steps also improve advanced variants like 2D PCA and Diagonal PCA, suggesting that input feature quality is often the primary bottleneck in recognition.

Conclusion and Future Work

The study confirms that targeted pre-processing can significantly bridge the gap between simple appearance-based models and complex real-world data. However, limitations remain: the system is currently restricted to static signs and requires a plain background for effective segmentation. Future work will focus on making pre-processing thresholds adaptive to automatically adjust to varying lighting conditions and extending the methodology to dynamic signs.

Summary updated

BibTeX
@inproceedings{suraj2007appearance,
  author = {M. G. Suraj and D. S. Guru},
  title = {{Appearance Based Recognition Methodology for Recognising Fingerspelling Alphabets}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
}
Figure