Introduction
Fingerspelling—the use of individual hand gestures to spell out words letter-by-letter—is a critical component of American Sign Language (ASL), often used for technical terms, proper nouns, or emphasis. However, automating its recognition in "in the wild" conditions is notoriously difficult due to visual ambiguity and environmental factors like motion blur and varying backgrounds.
Researchers have developed a new Transformer-based model that significantly improves the automatic recognition of ASL fingerspelling in real-world videos. By combining a fine-grained visual attention mechanism with optical flow data, the model achieves state-of-the-art accuracy while being nearly 10 times faster than previous iterative approaches.
Methods
The authors propose a sequence-to-sequence architecture that moves away from recurrent networks in favor of the Transformer architecture. The pipeline consists of several key stages:
- Feature Extraction: A pre-trained ResNet18 extracts spatial features from video frames.
- Context-Based Attention Module: This core innovation uses optical flow as a "prior" to guide the attention mechanism, focusing on relevant hand parts during rapid movement.
- Transformer Encoder: A multi-head attention layer captures long-range temporal dependencies better than traditional LSTMs.
- Joint Loss Function: The model is trained using Connectionist Temporal Classification (CTC) loss and Maximum Entropy Loss (MEL) to prevent overconfidence and improve generalization.
Experimental Setup
The study utilizes the ChicagoFSWild dataset, which contains 7,304 fingerspelling sequences from 192 unique signers in uncontrolled environments. This dataset serves as a benchmark for "in the wild" performance. Preprocessing evaluations showed that cropping the video to the "Face ROI" (Region of Interest) significantly boosts performance compared to using the whole frame.
Key Results and Insights
The proposed model sets a new benchmark for the ChicagoFSWild dataset, achieving a letter accuracy of 48.36% on the test set.
Key findings include:
- Efficiency: The Transformer-based approach allows for parallel training, reducing average processing time from 65ms to 6ms per frame.
- Motion as a Guide: Incorporating optical flow helps resolve ambiguity during fast transitions by treating change in hand position as informative data.
- Decoding Impact: Using Beam Search with a Language Model (BS+LM) provided the best results by leveraging linguistic context.
Conclusion and Future Work
While the model outperforms previous state-of-the-art approaches, a ~48% accuracy rate indicates that "in the wild" recognition remains an open challenge. Future work will focus on scaling the architecture to larger datasets like ChicagoFSWild+, implementing end-to-end segmentation, and integrating 3D hand pose estimation for more robust feature extraction.
Summary updated