Abstract Sign language serves as a crucial medium of communication for individuals with hearing and speech impairments, yet it presents barriers for those unfamiliar with its nuances. Recent advancements in artificial intelligence, computer vision, and deep learning have paved the way for innovative sign language recognition (SLR) systems. This paper provides a comprehensive survey of cutting-edge approaches to real-time sign language recognition, with a specific focus on Indian Sign Language (ISL). Various methodologies, including Convolutional Neural Networks (CNNs), Hidden Markov Models (HMMs), and transfer learning, are explored to evaluate their performance in recognizing static hand poses and dynamic gestures. Systems leveraging grid-based features, skin color segmentation, and feature extraction techniques such as Histogram of Oriented Gradients (HOG) are examined for their accuracy and efficiency. Furthermore, the integration of advanced frameworks, including pre-trained models like ResNet-50 and novel pipelines combining gesture reclassification with text-to-gesture synthesis, is discussed. This survey emphasizes the role of machine learning models, from Random Forests to state-of-the-art Large Language Models (LLMs), in addressing linguistic variability and enabling cross-linguistic translations between sign languages. By consolidating these diverse techniques, this study aims to provide valuable insights into the development of robust, real-time SLR systems that enhance accessibility, social integration, and inclusivity for the hearing-impaired community.
Introduction: The Communication Gap
Despite being the primary mode of communication for a massive population in India, Indian Sign Language (ISL) remains poorly understood by the general public. Traditional solutions, such as human interpreters, are expensive and not available in real-time. Early technological attempts relied on "data gloves" or wearable sensors, which are cumbersome, expensive, and socially stigmatizing. The challenge addressed in this review is the development of a portable, non-invasive, and highly accurate system—ideally via a smartphone camera—that can translate ISL gestures into text or speech in real-world environments.
Methodologies in ISL Recognition
The paper categorizes a wide array of methodologies used in recent years, transitioning from traditional computer vision to advanced deep learning:
- Feature Extraction: Traditional techniques like Histogram of Oriented Gradients (HOG) and skin color segmentation (using YCbCr or HSV color spaces) are discussed alongside modern landmark detection tools like MediaPipe.
- Deep Learning Architectures: The survey covers the use of Convolutional Neural Networks (CNNs) for static hand poses and Recurrent Neural Networks (RNNs) or Hidden Markov Models (HMMs) for temporal, dynamic gestures.
- Advanced Frameworks: More recent studies leverage Transfer Learning (using pre-trained models like ResNet-50, VGG-16, and AlexNet) and Graph Convolutional Neural Networks (G-CNNs) to capture the spatial relationships between hand joints.
- Sequence Modeling: The paper highlights the shift toward Transformer Networks and Large Language Models (LLMs) to handle the linguistic nuances and "video-to-text" translation of continuous signing.
Datasets and Benchmarks
While the paper is a review and doesn't introduce a single new dataset, it references several benchmarks used across the field:
- Custom ISL Datasets: Various studies mentioned trained on sets of 33 distinct hand poses (10 digits and 23 letters).
- International Benchmarks: For comparative analysis, the authors cite results from the KArSL-190 and KArSL-502 (Arabic) and LSA64 (Argentinian) datasets.
- The Data Gap: A recurring theme in the paper is the lack of large-scale, heterogeneous, and culturally inclusive datasets specifically for ISL compared to American Sign Language (ASL).
Key Insights: The Shift to Vision and Transformers
- From Sensors to Vision: The field has decisively moved away from hardware-heavy solutions (gloves) toward "vision-based" systems that use standard RGB or Depth cameras, making the technology more accessible.
- Isolated vs. Continuous Recognition: Most current success is found in "isolated" sign recognition (single letters or words). The major breakthrough in recent research is the application of Transformers to solve "continuous" sign language recognition, which treats signing as a sequence of data similar to natural language processing.
- Multimodal Fusion: Combining image data with hand landmarks (skeleton tracking) significantly improves robustness against noisy backgrounds and varying lighting conditions.
- LLM Integration: Beyond just identifying a gesture, researchers are beginning to use LLMs to translate "glosses" (the raw signs) into grammatically correct, natural language sentences.
Performance and Results
The paper cites impressive quantitative milestones from the surveyed literature:
- High Accuracy: Many CNN-based systems for ISL now exceed 90% accuracy.
- State-of-the-Art Peaks: One specific study (Bansal et al.) achieved a remarkable 99.82% accuracy for static gestures using modified AlexNet and VGG-16 architectures.
- Ensemble Success: Hybrid models and ensemble learning frameworks have demonstrated accuracies around 98.46% for isolated signs.
- Real-Time Viability: Several reviewed systems are now capable of running on Android platforms, processing gestures via smartphone cameras in real-time.
Challenges and Future Directions
Despite the high accuracy numbers, several "last mile" hurdles remain:
- Signer Independence: Many models perform well on the people they were trained on but struggle with new users. Developing models that generalize across different hand shapes and signing styles is a priority.
- Environmental Robustness: Lighting variability, background clutter, and "occlusion" (where one hand hides the other) remain significant technical obstacles.
- Computational Complexity: Many high-performing deep learning models are too "heavy" for mobile devices. Future work must focus on lightweight, edge-optimized architectures.
- Cultural Inclusivity: There is a pressing need for more diverse datasets that account for regional dialects and variations within ISL.
This summary is based on research published in the International Journal for Research in Applied Science & Engineering Technology (IJRASET), Volume 12, Issue XI, November 2024.
Summary updated