Abstract
Fingerspelling enables signers to represent proper nouns and technical terms letter-by-letter using manual alphabets, yet remains severely under-resourced for Indian Sign Language (ISL). We present the first continuous fingerspelling dataset for ISL, extracted from the ISH News YouTube channel, in which fingerspelling is accompanied by synchronized on-screen text cues. The dataset comprises 1,308 segments from 499 videos, totaling 70.85 minutes and 14,814 characters, with aligned video-text pairs capturing authentic coarticulation patterns. We validated the dataset quality through annotation using a proficient ISL interpreter, achieving a 90.67% exact match rate for 150 samples. We further established baseline recognition benchmarks using a ByT5-small encoder-decoder model, which attains 82.91% Character Error Rate after finetuning. This resource supports multiple downstream tasks, including fingerspelling transcription, temporal localization, and sign generation.
Introduction
While sign language recognition has advanced significantly, research often focuses on isolated signs or full-sentence translation. Fingerspelling—spelling out words letter-by-letter—is a critical "bridge" mechanism for communicating names, places, and technical jargon that lack dedicated signs. In Indian Sign Language (ISL), fingerspelling is particularly complex because it frequently employs two-handed configurations, unlike the one-handed system used in American Sign Language (ASL).
Existing ISL resources are largely composed of static images of single handshapes. This limitation makes it impossible to model co-articulation—the fluid way one letter's handshape blends into the next during rapid, natural signing. Without continuous video data, building effective real-world transcription tools for the Deaf community in India remains a significant challenge.
The Continuous ISL Fingerspelling Dataset
To address this data scarcity, researchers developed a multi-stage pipeline to create the first continuous ISL fingerspelling dataset. They leveraged the unique visual style of the ISH News YouTube channel, where fingerspelled words are presented alongside synchronized on-screen text cues. This provided a natural source of ground-truth labels for authentic, co-articulated signing.
The resulting dataset includes 1,308 segments from 499 videos, capturing over 14,000 characters. To prepare the data for modeling, the team used YOLOv8 for signer detection and MediaPipe Holistic to extract 75 body and hand keypoints at 15Hz. This approach ensures the models focus on the signer's movements rather than background noise or text overlays.
Experimental Results and Insights
The researchers established a baseline using a ByT5-small encoder-decoder transformer. As a byte-level model, ByT5 is uniquely suited for fingerspelling because it operates on individual characters, allowing it to handle the out-of-vocabulary proper nouns typical of spelled words.
Key findings from the experiments include:
- Specialized Data is Essential: A model pretrained on general ISL (the iSign dataset) failed completely on fingerspelling tasks, yielding a Character Error Rate (CER) over 400%. This proves that fingerspelling is a distinct linguistic mode that cannot be mastered through general sign language data alone.
- Baseline Performance: After fine-tuning on the new dataset, the ByT5 model achieved a CER of 82.91%. While this is high compared to the ~10% CER seen in mature ASL models, it represents an 80.8% relative reduction in error compared to the zero-shot baseline.
- Two-Handed Complexity: The results highlight the inherent difficulty of ISL's two-handed system, which involves more frequent occlusions and complex spatial relationships than one-handed systems.
Conclusion and Future Work
This dataset provides a vital benchmark for ISL accessibility tools. However, the authors acknowledge limitations, including a small pool of signers (3 individuals) and a vocabulary biased toward news terminology. Future research will focus on increasing signer diversity and exploring transfer learning from larger ASL datasets to help overcome the data scarcity challenges inherent in Indian Sign Language research.
Summary updated