Abstract
Sign language is crucial for the Deaf and Hard-of-Hearing community because it facilitates visual movement-based communication. Nevertheless, most are not familiar with it, rendering interactions with the hearing impaired complicated. While there has been significant work on languages, for instance, American and Chinese Sign Language, Pakistani Sign Language (PSL) at the word level has received less attention and has been studied based on static images. To address this, we introduce a deep spatiotemporal network for word-level PSL recognition from video. It commences by employing top-k frame extraction to enhance processing efficiency. Second, the ResNet-101 model is utilized for extracting deep spatial features from each frame. Subsequently, we introduce the Adaptive Motion Binary Pattern (AMBP), a new spatiotemporal feature descriptor that effectively extracts the spatiotemporal features. These spatial and spatiotemporal features are fused and input into the transformer model that processes these representations for better recognition. Experimental evaluations confirm that our framework achieves state-of-the-art results.
Introduction
While sign languages like American (ASL) or Chinese (CSL) have seen significant research, Pakistani Sign Language (PSL) remains relatively underexplored. Most existing PSL studies focus on "static" recognition—identifying individual letters or numbers from still images. However, natural communication is dynamic, involving sequential hand movements and temporal dependencies that static models cannot capture. The few existing dynamic PSL models often struggle with environmental noise, lighting variations, and limited vocabularies. This paper aims to bridge this gap by providing a robust framework for word-level recognition that can handle the complexities of real-world video data using a novel deep spatiotemporal network.
Proposed Methodology
The proposed framework follows an end-to-end pipeline designed to capture both the visual appearance and the motion dynamics of sign language gestures.
1. Top-K Keyframe Extraction
To improve computational efficiency, the system identifies the 30 most "informative" frames by calculating contrast differences between consecutive frames. This step effectively reduces redundancy while preserving the core visual information necessary for recognition.
2. Deep Spatial Feature Extraction
Each selected frame is processed through a pre-trained ResNet-101 model. Features are extracted from the Global Average Pooling (GAP) layer, resulting in a 2048-dimensional vector representing the spatial characteristics of the image, such as hand shapes and positions.
3. Adaptive Motion Binary Pattern (AMBP)
AMBP is the paper's primary innovation—a temporal descriptor that analyzes three consecutive frames. It uses a dynamic threshold based on pixel intensity differences to remain robust against lighting changes and noise. By encoding motion using bitwise XOR operations, it produces a 256-dimensional vector representing the temporal dynamics of the movement.
4. Transformer-based Sequence Modeling
The spatial and temporal features are fused into a 2304-dimensional vector per frame and fed into a Transformer. The Transformer utilizes 1D convolutions for embedding and sinusoidal positional encoding to learn the sequential relationships of the signs over time, effectively handling long-range dependencies.
Experimental Setup and Datasets
The model was evaluated on two distinct datasets to test its robustness:
- PkSLMNM Dataset: Contains 665 videos of 180 individuals performing 7 classes of adjectives (e.g., "bad," "best," "scared").
- PSL Dictionary Dataset: A more diverse set consisting of 160 samples covering 80 different words.
Results and Analysis
The proposed framework outperformed several well-known video action recognition models, including C3D, TSM, and I3D. On the PkSLMNM dataset, the model achieved an accuracy of 85.7%, surpassing the previous state-of-the-art (SLATN) at 82.66%. On the PSL Dictionary dataset, it reached 81.25% accuracy.
Key Insights
- Hybrid Feature Fusion: Combining deep learning features (ResNet) with handcrafted-inspired temporal descriptors (AMBP) provides a finer understanding of movement dynamics than either method alone.
- Ablation Study: Removing the AMBP component dropped the PkSLMNM accuracy from 85.7% to 77.4%, proving that the temporal descriptor is vital for performance.
- Efficiency: The study found that 30 keyframes provided the optimal balance for accuracy; adding more frames led to a slight decrease in performance due to increased noise and redundancy.
Conclusion
This research presents a significant advancement in word-level Pakistani Sign Language recognition. By integrating spatial features from ResNet-101 with the novel AMBP temporal descriptor and a Transformer model, the system achieves state-of-the-art results. Future work will focus on optimizing the model's efficiency to enable real-time PSL translation on mobile and wearable devices, expanding the vocabulary to cover a broader range of natural communication.
Summary updated